Apache Airflow 3, a scheduler that knows what depends on what

Apache Airflow 3, a scheduler that knows what depends on what

Recurring jobs can live in system scheduler entries while there are five of them and none depends on another. With fifty jobs, one of which must wait for three others, and where a failure means knowing which data is now stale, a different mechanism is needed.

Airflow describes a pipeline as a dependency graph: tasks, order, and conditions. The platform tracks what ran, what failed, and what waits, and the interface shows it in a form readable during an outage.

Version three, released in April 2025, brought changes that alter how you work more than the number suggests.

Your first pipeline

from airflow.sdk import dag, task
from datetime import datetime

@dag(schedule="0 6 * * *", start_date=datetime(2026, 1, 1), catchup=False)
def daily_report():

    @task
    def fetch_orders() -> list[dict]:
        return db.orders_from_yesterday()

    @task
    def compute_metrics(orders: list[dict]) -> dict:
        return {"count": len(orders), "value": sum(o["amount"] for o in orders)}

    @task
    def send_report(metrics: dict) -> None:
        mail.send("board@company.com", metrics)

    send_report(compute_metrics(fetch_orders()))

daily_report()

Dependencies follow from passing results between tasks, so they need no separate declaration. The last line describes the whole graph: fetch, compute, send, in that order.

The namespace used in the import is new in version three. Previously you imported from various internal places, which caused compatibility breaking changes on upgrades. Now there is one set of stable interfaces for authoring pipelines, separated from the platform's internals.

Catchup is off by default in version three, since catchup_by_default now defaults to false. The example still states it explicitly, so the intent is visible in the code rather than hidden in an installation's configuration. If you turn it on deliberately, or carry over a configuration from version two where it defaulted to on, a pipeline with a start date a year back fires three hundred sixty five runs at once.

Pipeline versioning

The most important change in version three and the answer to a problem that previously caused the most confusion.

In earlier versions, rerunning an old run executed it against current code. A run from a month ago, resumed after a logic change, produced a different result from the original, and nobody knew which was correct.

Now the platform tracks the code version a run started with and by default resumes it against the same one. A setting lets you choose the behaviour, but the default is the right one: a March run replays as it looked in March.

That changes how historical data is corrected, with one exception that is easy to forget. Backfills kept the old behaviour and run against the latest code by default. The rerun_with_latest_version setting decides it: false for clearing and rerunning, true for backfills, and overridable per request or per pipeline.

The interface shows which version each run executed against. When diagnosing a data discrepancy that is the first thing to check, and previously it had to be hunted in repository history.

Assets and event driven triggers

The second large change concerns what starts a pipeline. A time schedule is simple and carries one flaw: a job at seven assumes the previous step's data is ready, and if it is not, it processes stale data.

The asset based model inverts that logic. A pipeline declares what it produces and what it consumes, and the platform runs it once the needed data appears.

from airflow.sdk import asset

@asset(schedule="@daily")
def raw_orders() -> None:
    db.load_from_source()

@asset(schedule=raw_orders)
def clean_orders() -> None:
    db.transform()

The second pipeline has no time of its own. It runs once the first finishes and updates its asset, so there is no interval to guess and no safety margin to add.

Version three extended that with triggers from events outside the platform. A queue message, a file in object storage, or an event from an external system can start a pipeline, removing the commonest workaround of polling a source every five minutes.

Tasks in other languages

Version three added a layer allowing individual tasks to be written in languages other than Python while the pipeline description stays in Python.

That answers a recurring situation in data teams: processing logic already exists in another language, and the only reason to rewrite it in Python was a tool limitation. Now a task can run where it lives, with the platform responsible for order, retries, and visibility.

The practical conclusion: with existing code in another language, check that route before planning a migration. Rewriting working code purely to fit a tool is rarely a good investment. The coordinator layer and the Java and Go SDKs are marked experimental as of version 3.3, so treat them as a route to evaluate rather than a production foundation.

Retries and error handling

A data pipeline fails regularly, since sources go down and data arrives incomplete. How those situations are handled separates a production pipeline from a script.

@task(retries=3, retry_delay=timedelta(minutes=5), retry_exponential_backoff=True)
def fetch_from_api() -> list[dict]:
    return client.fetch_data()

Retries with growing delay handle transient errors, meaning service unavailability and rate limits. That covers most failures which repair themselves.

A separate matter is what to do once retries run out. By default the task is marked failed and dependent tasks wait. In a pipeline where one failed step should not block the rest, a different trigger rule for the next task can be set.

Distinguish a failure from an absence of data too. A task that found nothing to process is not a failure, and marking it skipped rather than failed is better, otherwise alerts turn into noise.

The last thing is notification. A failed pipeline without one is noticed when somebody asks about a missing report. A failure callback sending a message to a team channel costs a few lines and cuts reaction time from days to minutes.

Testing pipelines

A pipeline is code, so the same rules apply as everywhere else, and yet it is the least tested part.

The simplest and most effective move is extracting processing logic outside the task definition. A function taking data and returning a result is testable with an ordinary unit test, without running anything.

def compute_metrics(orders: list[dict]) -> dict:
    return {"count": len(orders), "value": sum(o["amount"] for o in orders)}

@task
def metrics_task(orders: list[dict]) -> dict:
    return compute_metrics(orders)

The task becomes a thin wrapper and all the logic lives somewhere checkable in milliseconds. That one change delivers the most and needs no tooling beyond what the team already uses.

The second level checks the pipeline definitions themselves: whether they all parse without error, whether the graph holds no cycles, whether identifiers are unique. Such a test in the build pipeline catches typos before they reach production.

The third is running a single task against test data. It helps with tasks touching external systems, where a unit test falls short and running the whole graph costs too much.

Worth knowing where that gymnastics comes from. The pipeline definition here is a separate graph, so running anything in a test needs a scheduler and a metadata database, which is why the logic gets pulled outside in the first place. Tools assembling a flow from decorators on ordinary functions, such as Prefect, do not have that problem, since the flow is called in a test like any other function. The price runs the other way: there the graph structure is known only during execution rather than before the start.

Running and maintaining it

The platform consists of several parts: a scheduler deciding what to run, an interface server, a metadata database, and workers. That is considerably more than a single service and worth knowing before deciding.

Self hosting means maintaining those parts, backing up the metadata database, and ensuring workers have enough resources. For one team that is a few hours a month; for a larger installation it is a role of its own.

The alternative is managed services, offered by the major cloud providers and by companies specialising in this platform. You then pay for maintenance and gain time, losing some control over version and configuration.

The third route is skipping the platform for simple needs. Five independent jobs need nothing beyond a scheduler, and deploying this platform for such a set is disproportionate.

Migrating from version two

Moving to version three is feasible and does not come down to bumping a number in dependencies.

The largest change is imports. Pipeline definitions reaching directly into the platform's internals must move to the new namespace. Across a dozen or so pipelines that is an hour's work; across two hundred it is worth scripting the commonest patterns.

The second is renaming. The dataset concept was replaced by assets while keeping the same inlet and outlet mechanism. Code using the old name needs fixing, though the behaviour stays close.

The third is rerun behaviour. The previous version always used current code; the current one returns to the original version by default. That is usually an improvement, but if your data correction process relied on the old behaviour, set it deliberately.

The practical order: first run a test installation on the new version and pass a copy of your pipelines through it, then fix whatever fails to parse, and finally plan the production switch. Version three preserved backward compatibility for most definitions, so the fix list is usually shorter than a major version bump suggests.

Airflow against the alternatives

The third row is a different category, though people confuse them. This platform schedules repeated data processing at set times or after events. A durable execution solution handles business processes that run once per order and last weeks. The overlap is small.

Choosing between the first two rows depends on whether you start from scratch. With an existing installation and hundreds of pipelines, switching is costly and rarely justified. On a new project compare both, since the conceptual models differ considerably.

The argument usually settling it in this platform's favour is the number of ready integrations. A connection to databases, data warehouses, cloud services, and queueing systems usually exists and is used by many teams, which means somebody else has already found the edge cases. With less popular tools that work falls to you.

The second argument is the availability of people. Experience with this platform is widespread, so a new person on the team usually knows the basics. That sounds like a soft argument, and over several years of maintaining an installation it carries practical weight.

Common mistakes

The first is catchup switched on by hand or inherited from a version two configuration. A pipeline with a start date a year back generates hundreds of runs at once and blocks everything else, though version three takes deliberate effort to get there, since the default is off.

The second is processing logic in the pipeline definition file. The scheduler parses that file many times a minute, so code querying a database at import time slows the whole installation.

The third is tasks without timeouts. A task hanging indefinitely occupies a slot and blocks others while reporting nothing.

The fourth is passing large data between tasks. Results travel through the metadata database, so passing a table rather than a path to it clogs the database and slows everything.

The fifth is one pipeline covering an entire process. Thirty tasks in one graph make retrying a fragment hard and mean a failure at the end requires repeating everything.

The sixth is skipping tests. A pipeline is code, so extract processing logic into functions with tests rather than checking it only by running the whole graph.

FAQ

What is Airflow for?

Scheduling and running data processing pipelines with dependencies between steps. The platform handles order, retries, and state visibility, and you describe a pipeline in Python code as a task graph.

What did version three bring?

Three things change how you work: pipeline versioning, so an old run replays against the code of its time; an asset based model with event triggers rather than schedules alone; and a stable namespace for authoring pipelines, separated from the platform's internals.

Does Airflow suit small projects?

Rarely. The platform needs a metadata database, a scheduler, an interface server, and workers, so with five independent jobs the maintenance cost exceeds the benefit. It starts making sense with dependencies between tasks and a need for visibility, usually from a dozen or so pipelines up.

How does it differ from Temporal?

Different purpose, despite similar sounding descriptions. Airflow schedules repeated data processing, run on a cycle or after an event. Temporal executes business processes started individually, which may last weeks and need resilience to failure midway.

Must tasks be in Python?

The pipeline description must, but version three allows individual tasks in other languages. That helps when processing logic already exists and rewriting it would be work done purely to fit a tool.

Documentation sits on the project site, and the version three changes appear in the release announcement.