Prefect 3, Python flows and the hybrid model

Prefect 3, Python flows and the hybrid model

Prefect is a tool for orchestrating workflows: running tasks in a defined order, retrying them on failure, scheduling them in time, and showing what happened. The late July 2026 release carries version 3.8, the licence is Apache 2.0, and required Python spans 3.10 to 3.14.

Two things separate it from tools in this category enough to start with them: how a flow gets described and how control gets separated from execution.

A flow is a function, not a graph

In the classic approach you describe a task graph in a separate file, usually in a format the tool imposes, and only then does that graph call your code. Here there is no separate file and no separate format.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=30)
def fetch_data(source: str) -> list[dict]:
    return client.fetch(source)

@task
def transform(rows: list[dict]) -> list[dict]:
    return [normalise(r) for r in rows]

@flow(name="daily-import")
def run_import(sources: list[str]):
    for source in sources:
        data = fetch_data(source)
        transform(data)

A decorator turns an ordinary function into a task and another into a flow. Structure follows from how the functions call each other, so a loop is a loop, a condition a condition, and an exception an exception.

The practical consequence runs deeper than it looks. A flow can be called in a test like an ordinary function, with no scheduler running and no environment. A task can be tested separately. The code reads for somebody who does not know this tool.

The price is that a flow's structure is known only during execution. With a graph described up front you know before starting what will happen; here you know afterwards. With branching driven by data that is an advantage; where you need to show a plan before running, rather a drawback.

The hybrid model

This is the second thing worth understanding and the main argument in a conversation with a security department.

The control layer, meaning scheduling, state tracking, and the interface, can run in the vendor's cloud. Execution happens on your infrastructure: your cluster, your machine, your cloud account.

That means data does not pass through the vendor. What reaches the control layer is metadata: task name, duration, state, error message. The content of the processed records stays where it was.

That difference settles situations where an external orchestrator would normally not pass review. A team processing personal or financial data can use a managed control layer, since that layer never sees the data.

Know where the boundary actually runs in practice, though. An error message from an exception containing a fragment of data reaches the control layer along with the stack trace. So does a log line your code writes. That is where care is needed, since the default behaviour can carry more across than you assume.

Work pools and running things

A work pool describes where flows should execute: a container cluster, a cloud service running containers, a serverless environment, or an ordinary process on a machine.

Separating a flow's definition from its execution location is practical here. The same code runs locally during development and on a cluster in production, with nothing changed in the flow itself. What changes is the pool, not the code.

Understanding how that works underneath matters for diagnosis. A worker process polls the control layer for tasks to run, fetches them, and executes them. A flow that does not start usually has one of three causes: no worker process is running, the running one serves a different pool, or it cannot reach the flow's code.

That last one surprises most often. The control layer knows a flow should run and does not hold its code. The code has to be reachable by the worker: in a container image, in a repository fetched at start, or in file storage. Configuring that is a separate step, easily forgotten on a first deployment.

Retries, state, and what an orchestrator provides

Worth naming what you actually pay for here, since a loop over tasks can be written yourself in half a day.

Retries with a delay are a declaration on a task rather than code inside it. A task reaching an external interface that is sometimes unavailable gets three attempts spaced out, and that is one line.

Result caching lets you skip a task whose result is already known. For a flow interrupted midway, rerunning does not repeat steps that succeeded, provided their input has not changed.

Concurrency gets limited declaratively. A task reaching a database that cannot take a hundred parallel connections gets a limit, and the orchestrator enforces it regardless of how many flows run at once.

State visibility is what a self built solution lacks most. The question of whether the nightly import succeeded has an answer on screen here, while with a script in a system scheduler it means digging through logs.

The last item is notification. A rule stating that a failed flow should message a given channel is configuration rather than code inside an exception handler.

Deployments and schedules

A flow written in an editor is not yet something that runs by itself. It has to be deployed, meaning telling the control layer where the code lives, when it should start, and in which pool.

A deployment gets described in a configuration file kept in the repository beside the code. That is the right approach, since the definition passes review like any other change, and reproducing the environment on a new account reduces to one command.

A schedule can be a time expression, an interval, or a rule accounting for time zones and non working days. That last one gets undervalued until a nightly import starts an hour earlier after a clock change and lands inside the database maintenance window.

Beyond a schedule, a flow can start from an event: a file appearing, an interface call, or another flow finishing. That last variant replaces the artificial delays people normally use to chain jobs in a system scheduler, and is one of the reasons for reaching for an orchestrator at all.

Decide what happens when runs overlap too. A flow scheduled every five minutes that occasionally takes seven will have two runs at once, and when both write to the same table that is usually not what was intended. A concurrency limit at the deployment level settles it with one declaration.

Observability and diagnosis

Recording state is the foundation, and over longer use a few more things help.

Tags applied to flows and tasks let you filter runs by area, team, or environment. Without them a run list after a month is a wall of entries with no way to navigate it.

Artefacts let you attach a table, a chart, or a note to a run, visible in the interface. For data processing that is a convenient home for a report: how many records came in, how many were rejected, and why. Somebody checking whether an import succeeded gets an answer without opening logs.

Understanding the transitional states matters, since diagnosis often reduces to telling two similar situations apart. A flow waiting for a free slot in a pool looks much like a flow nobody picked up, while the causes differ entirely and get fixed in different places.

For applications built on language models it pays to pair this record with model call tracing, through Arize Phoenix for instance. The orchestrator shows that a task ran; the tracing tool shows what went to the model and what came back. One without the other leaves a gap.

Prefect against the alternatives

The last row deserves honest consideration, since it gets skipped for reasons that are not technical. Three jobs run once a day with a failure notification need no orchestrator. Introducing one means another service to maintain and another place where something can fail.

The boundary runs at dependencies between tasks and at needing to know what happened. Once task B must wait for A, and on a failure in C you must know which records got through, a system scheduler stops sufficing.

The third row solves a different problem, and confusing them is common. There the point is processes lasting days, with a guarantee that state survives a process failure. Here it is repeatable data processing with visibility and retries.

The fourth row moves along the same axis as the first, in the opposite direction. A flow described in YAML reads for somebody who does not write Python, and the tasks themselves can be in any language, which on a mixed team often matters more than the convenience of testing a function. The price is double: elaborate control logic inside the document turns illegible faster than the line count suggests, and a handful of plugins are reserved for the commercial edition.

Pricing and self hosting

The licence is open, so a full self hosted deployment is available at no charge. You need a control layer, a database, PostgreSQL for instance, and at least one worker process.

The managed variant carries a free tier called Hobby, capped at two users and five deployments. Once you hit the fifth deployment you cannot create another until you delete an existing one or move to a paid plan, so it is a hard ceiling rather than a soft threshold with an overage charge.

The paid plans come in two different rate shapes, and that is easy to miss on a quick glance at the price list. Starter costs 100 dollars a month for the whole account, with three users and twenty deployments. Team is also 100 dollars, but per user per month, with four included at the entry point, so the bill grows with the team rather than staying put. Pro and Enterprise are custom priced and billed annually. Check the vendor's current price list, since thresholds and limits change more often than the architecture does.

More important than the rate is the billing principle. You pay for team seats and workspaces rather than task executions or consumed compute, which for frequently running flows changes the bill fundamentally against tools metering per run. An application executing tens of thousands of tasks a day costs no more here than one executing hundreds. The ceiling sits elsewhere: in the user and deployment counts attached to a plan, from five deployments on the free tier through twenty and a hundred to a thousand on Pro. Run retention deserves a separate look, since on the two lowest plans the history disappears after seven days.

When choosing between self hosted and managed, cost out the operational work. The control layer is a service that must run, carry backups, and stay updated, and its failure halts scheduling for every flow. For a team with nobody dedicated to infrastructure the managed variant usually comes out cheaper than a rate comparison suggests.

Common mistakes

The first is a worker with no access to the flow's code. The control layer knows a flow should start and has nothing to run, and the symptom looks like a task stuck in a queue.

The second is a worker serving a different pool from the one the task landed in. Everything is configured correctly and nothing happens.

The third is carrying data into the control layer through logs and error messages. The hybrid model protects processed data rather than what you write into a log yourself.

The fourth is no concurrency limit on tasks reaching a shared resource. A hundred parallel tasks hitting one database can bring it down, and the orchestrator will happily launch them.

The fifth is result caching without thinking about the key. A task considered done because its input looks the same skips processing data that changed meanwhile.

The sixth is introducing an orchestrator for three jobs run once a day. You gain a service to maintain and visibility nobody needs at that scale.

FAQ

How does Prefect differ from Airflow?

In how a flow gets described. Here a flow is an ordinary function with a decorator, so loops and conditions are loops and conditions, and testing reduces to calling a function. Airflow describes the graph separately, which gives knowledge of the structure before running at the cost of more to learn.

What is the hybrid model?

A separation of control from execution. Scheduling, state tracking, and the interface can run in the vendor's cloud while tasks execute on your infrastructure, so data does not pass through the vendor. What reaches the control layer is metadata rather than the content of processed records.

Can I run the whole thing myself?

Yes, the licence is open and a full self hosted deployment is available at no charge. You need a control layer, a database, and at least one worker process, and in exchange you take on backups and updates for that service.

Why is my flow stuck in a queue?

Usually for one of three reasons: no worker process is running, the running one serves a different pool, or it cannot reach the flow's code. That last one surprises most, since the control layer knows the schedule and stores no code.

How is the managed variant billed?

By team seats and workspaces rather than task executions, so the run count stops affecting the price. Watch the shape of the rate: the Starter plan is 100 dollars a month for the whole account, while the Team plan is 100 dollars a month per user. Every plan also carries its own deployment cap, from five on the free tier to a thousand on Pro.

Documentation sits on the project site, and the code and releases in the GitHub repository.