Kestra, flows in YAML and many languages

Kestra, flows in YAML and many languages

Kestra is a workflow orchestration platform where a flow's definition is a YAML document and tasks execute in any language. The current release line is 1.3, open since March 2026, and the engine sits under the Apache 2.0 licence.

That single design decision separates it from tools where a flow is code in a particular language, and it drives every other difference. Understanding it before comparing anything else pays.

A flow as a document rather than code

In tools built around one language a flow is a function or a class, so reading it requires knowing that language. Here a flow is a description of structure, with the code sitting inside the tasks.

id: daily-import
namespace: data.production

tasks:
  - id: fetch
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import requests
      data = requests.get("https://api.example.com/orders").json()
      print(len(data))

  - id: transform
    type: io.kestra.plugin.jdbc.postgresql.Query
    sql: INSERT INTO report SELECT * FROM staging WHERE day = CURRENT_DATE

triggers:
  - id: daily
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 3 * * *"

The structure is visible immediately: what follows what, what starts the whole thing, where the boundary between steps runs. The first task runs Python, the second executes a database query, and nobody needs to know both to understand the flow.

Two consequences follow, both real.

The first is organisational. A flow can be read by somebody who does not program in the task language: an analyst, an administrator, a person signing off the work. On mixed teams that sometimes matters more than code elegance.

The second is technical and less pleasant. Logic expressed in a document is poorer than logic in a programming language. A condition or a loop can be written, while elaborate control logic in YAML becomes illegible faster than the line count suggests.

Multiple languages as the core

Since the code sits inside tasks, language stops being a choice at the level of the whole tool.

One flow can run a Python script, a database query, a shell command, a container with a ready image, and a call to an external interface. Those are not workarounds but ordinary task types.

The gain shows in organisations where data passes through several teams. Analysts write queries, engineers write scripts, and the platform team runs containers, and all of it fits in one flow rather than in three systems chained by calls.

There are well over a thousand plugins, a figure worth noting since it determines how much of this you get ready made. Connecting to a typical database, cloud file storage, or a queueing system is usually an entry in the document rather than code you have to write.

Remember, though, that a plugin is a dependency like any other. For rarely used ones it pays to check when they were last updated, since a catalogue that size is not uniform in how well it is tended.

Events and triggers

A flow can be started by a schedule, and that is the obvious variant. The others are more interesting.

An event trigger reacts to a file appearing in storage, a message in a queue, a change in a database, or an interface call. That way a flow starts when there is something to do rather than when a deadline arrives.

A trigger based on another flow finishing lets you build chains with no artificial delays. That is the thing people solve with a system scheduler by setting the second job fifteen minutes later and hoping the first finished.

Know the polling trigger separately, since it carries a property that surprises when costing out load. It checks a source at set intervals regardless of whether anything changed, so across several such flows the system works constantly while nothing happens.

When designing, start with the question of what should actually start the flow. The answer "daily at three" is often a simplification of something that really means "once yesterday's data arrives", and those two behave differently when the data runs late.

Open and commercial editions

The split is clear here and deserves knowing before a company deployment.

The open version under Apache 2.0 covers the engine, the interface, and most of the plugins, meaning everything needed to run flows. This is not a version cut down for demonstration but a working tool. The plugin catalogue does split in two, though, and a handful of entries marked as enterprise require the commercial edition, so check that before committing to an unusual integration.

The commercial version adds things an organisation needs rather than a team: logging in through a corporate identity system, role based permissions, support with response time guarantees, and mechanisms for managing several teams on one installation.

The split is not a two way one, though, since alongside a self hosted installation the vendor also runs a managed variant. Kestra Cloud is the same platform operated on the vendor side, with the enterprise plugins included and billing that grows with usage, while access is granted on request, so it is not a service you sign up for in five minutes.

The practical conclusion is simple. A team running flows for itself manages with the open version and loses nothing important. An organisation where five teams with different permissions share one installation needs access control, and that sits on the commercial side.

Check that boundary against your own case before deciding, since the list of features assigned to each edition changes between releases.

Passing data between tasks

This is where the document based approach differs from code the most, and it deserves understanding, since most mistakes arise here.

In code a function's result gets passed on as a value. Here tasks are isolated from each other, so one's output goes to storage and the next refers to it through an expression naming the task identifier and the output name.

The practical effect cuts both ways. Tasks can run in different languages and on different machines, since they share no memory. And: passing data carries a cost, since it travels through storage rather than a variable.

From that follows a rule worth adopting from the start. Pass identifiers and paths rather than contents. A task fetching a million records should write them to a database or a file and pass on a reference rather than the whole array. Passing large structures through storage works and stops working at a scale nobody expects.

The second thing is variables and expressions. The document supports templates letting you reach the run date, input parameters, and earlier task outputs. That is convenient and tempting to abuse: an expression with three nested conditions inside a text field is code written in the worst possible place.

The practical boundary reads: an expression selecting a value or composing a path is fine, while an expression making a business decision belongs in a task with a script that can be read and tested.

Error handling and retries

Worth describing what you get ready made, since that is the main reason for reaching for an orchestrator rather than a system scheduler.

Retries are declared on a task: attempt count, interval, and how it grows. A task reaching an external interface that is sometimes unavailable gets three attempts, and that is a few lines in the document rather than a loop in code.

Error handling tasks run when something went wrong and let you clean up or notify the right person. That differs from a situation where a flow simply stops and the information lands in a log nobody opens.

Concurrency limits work at flow level and at task level. A flow scheduled every five minutes that occasionally takes seven will, without that limit, have two runs at once, and when both write to the same table that is usually not what was intended.

The last item is handling delays and pauses. A task can wait a set period, can halt the flow until a condition holds, and in processes involving people can wait for approval. Waiting consumes no execution resources meanwhile, which in a self built solution would require durable state and a resumption mechanism.

Kestra against the alternatives

The comparison with the second row teaches the most here, since both tools solve the same problem by opposite means.

There a flow is a function, so it can be called in a test and the language's full power is available. Here a flow is a document, so anybody reads it and tasks can be in anything. The choice reduces to whether your constraint is logic complexity or diversity of technologies and people.

Practical guidance: for a Python team building data pipelines the second row usually wins on convenience. For an organisation where a flow touches a database, a container, a shell script, and three teams, the first wins on legibility.

The fourth row solves a different problem, and confusing them is common. There the point is processes lasting days with a guarantee of surviving failures; here it is repeatable processing with visibility and retries.

Running and maintaining it

Running it needs the server itself plus a database for state, usually PostgreSQL. At larger scale a queue and separate worker processes join that.

A deployment usually starts from a single container with an embedded database, which suffices to evaluate the tool within an hour. For production it pays to set up an external database immediately, since migrating state later is work nobody plans for.

Keep flows in a repository and deploy them automatically rather than editing them in the interface. The built in editor is convenient for experiments and tempting in production, while a change made through the interface passes no review and appears in no project history.

Settle the namespace split early too. At twenty flows everything looks fine regardless of structure, and at two hundred the absence of areas makes the list useless.

The last item is storing task outputs. A task's output goes to storage and stays available to later steps, which is convenient and grows over time. A policy for deleting old runs is mandatory here rather than optional.

Common mistakes

The first is expressing complicated control logic in the document. Conditions and loops work, while elaborate logic in YAML becomes illegible, and the right home then is a script inside a task.

The second is editing production flows in the interface. The change takes effect immediately, passes no review, and leaves no trace in project history.

The third is a schedule where an event belongs. A flow started at a fixed hour processes data that may not have arrived yet, and the failure looks random.

The fourth is many polling triggers. The system works constantly while nothing happens, and load grows linearly with the number of such flows.

The fifth is no policy for deleting old runs and outputs. Storage grows at a rate driven by data size rather than task count.

The sixth is a production installation on the embedded database. It works until the first restart under load, and moving state later is a project of its own.

The seventh is passing large data sets between tasks instead of identifiers. Everything travels through storage, so a solution that works at a thousand records stops working at a million, and with no clear warning threshold.

FAQ

How does Kestra differ from Prefect?

In how a flow gets described. Here a flow is a YAML document and tasks can be in any language, so somebody who does not program can read it too. In Prefect a flow is a Python function, which gives the language's full power and the ability to test by calling it.

Is the open version enough for production?

Yes, it covers the engine, the interface, and most of the plugins under Apache 2.0. The commercial version adds corporate identity login, role based permissions, enterprise only plugins, and support with response time guarantees, meaning things an organisation needs rather than a single team. A vendor managed variant, Kestra Cloud, exists separately.

Which languages do I write tasks in?

Any, since the code sits inside a task. One flow can run a Python script, a database query, a shell command, and a container, and there are well over a thousand plugins for typical systems.

Can this be versioned like code?

Yes, and it should be. Keep flows in a repository and deploy them automatically, since editing in the interface bypasses review and leaves no trace in project history. The built in editor suits experiments rather than production.

What do I need to run it?

A server and a database for state, in practice PostgreSQL. For evaluation a single container with an embedded database suffices, while for production it pays to set up an external database immediately, since moving state later is a task of its own.

Documentation sits on the project site, and releases in the GitHub repository.