What Workflow Orchestration Tools Ask of You, and What the Engine Does

A payment run stops halfway. Four hundred transfers are confirmed, the fifth hundred is in flight, and the process that was tracking which was which is gone: the pod was evicted during a node drain. Nothing is corrupt. Nothing is lost, exactly. It is simply that no surviving component knows which transfers happened, and the only way to find out is to go and ask the payment provider four hundred and something times.

That is the failure workflow orchestration exists to prevent, and it is worth stating plainly before any of the vocabulary arrives, because the vocabulary is where the subject gets slippery. Ask five engineers what is workflow orchestration and you will get five answers pitched at five different layers, which is itself a hint about how much the term is carrying.

What is workflow orchestration, in the vocabulary the engines use

Strip the marketing and a workflow orchestration system is a supervisor with a notebook. Your process is broken into steps. Before a step runs, the supervisor writes down that it is about to run; after it returns, the supervisor writes down what it returned. If the process dies, a new one starts, reads the notebook, and skips everything already written in it.

The notebook is the whole product. Everything else — the SDKs, the web console, the retry policies, the signal and query APIs — is scaffolding around a durable append-only log of “this step ran and produced this”. When a vendor says their workflow orchestration engine gives you reliability, this is the mechanism, and it is genuinely a good mechanism. Temporal, Restate, DBOS, AWS Step Functions and Azure Durable Functions all implement some version of it, and so does Cadence, the Uber project Temporal was forked from in 2019.

So the short answer to what is workflow orchestration is: a durable log, plus a rule about who is allowed to write to it. The important asymmetry follows from that: a workflow orchestration engine can only skip what it wrote down, and it can only write down what it was able to see.

The rewrite that workflow orchestration tools do not advertise

Here is where the fairy tale starts. A landing page for workflow orchestration tools shows a decorated function, six lines and one annotation, and the promise is that the six lines now survive anything. What the six lines do not show is that every outside effect in your program has to become a named, addressable, individually recorded thing before the engine can help.

Take the payment run. In its original form it is probably a loop with a database call, an HTTP call and a log line in the body. To put it under a workflow orchestrator you have to answer, for each of those, a question you were previously allowed to ignore: is this an activity or is it workflow code? Activities can fail and be retried and their results are recorded. Workflow code cannot do I/O at all, because it will be re-executed.

That distinction propagates outward. Your ORM session probably cannot live in workflow code. Your metrics client probably cannot either, at least not the part that reads a clock. random, uuid4, datetime.now, iteration over a set, a dictionary you assumed was ordered because it has been since CPython 3.7: all of it becomes suspect, because all of it can produce a different answer on the second pass. Temporal’s Python SDK ships a sandbox that reaches into the import system to make some of this loud rather than silent, which tells you how often it bites.

None of this is a criticism of the design. Replay is the cheapest known way to get durability without asking the programmer to serialise their own state by hand. It is a criticism of the sales pitch, which presents a constraint on how you may write programs as a decorator you may add to programs you already wrote.

Determinism is a property of your whole dependency tree

The rule as written is “workflow code must be deterministic”. The rule as experienced is “workflow code must be deterministic, and so must every library it transitively calls, including the ones you did not choose”.

This is the part that is genuinely hard to audit. A JSON serialiser that iterates a dict. A validation library that caches by object identity. A retry helper with jitter. A gRPC stub that attaches a deadline computed from the current time. Each is fine in ordinary code and each is a latent replay bug, and the failure does not show up when you write it. It shows up months later, during the restart, which is the exact moment you were counting on the engine.

Restate takes a different route here: instead of replaying your function body it journals each suspension point and resumes from the journal (the same move that makes a paused coroutine a unit of distributed work), which narrows the determinism requirement considerably. DBOS leans on Postgres and asks you to think in terms of transactions. The trade-offs differ enough that “which engine” is a real architectural decision and not a vendor preference. But every one of them puts some constraint on the shape of your code, and the constraint is the actual cost of adoption. Whichever workflow orchestrator you land on, budget for an audit of the dependency tree rather than an afternoon of adding decorators.

An aside on history size

Something that rarely comes up until it is a problem: the log is not free to keep. Each recorded step is a row, each row has an input and an output, and inputs and outputs are the arguments your code passes around. A workflow that fans out over ten thousand items and passes a modest payload to each of them writes a history that is measured in gigabytes, and the engine has to read that history back on every replay. Teams discover this by watching replay latency climb over a quarter. There are answers (continue-as-new, payload codecs, external blob storage for large arguments), and this piece is not going to resolve which one is right, because the answer depends on a retention policy that is usually someone else’s decision.

What the guarantee actually is

Worth being precise, because the words are used loosely.

An orchestration engine gives you: at-least-once execution of each recorded step, exactly-once observation of each step’s result from the workflow’s point of view, and durable progress across process restarts. It does not give you exactly-once side effects. If a step charges a card and the process dies after the charge but before the write, the engine will run that step again. The idempotency key is still your job. It has always been your job, and no workflow orchestration engine on the market takes it off you.

That is not a small caveat, and it is the one most consistently blurred. “Exactly once” on a marketing page means “exactly once as far as the log is concerned”. The payment provider has its own opinion.

So: use one or not

For a process with more than a handful of steps, real money or real outside effects, and a runtime measured in minutes or longer: yes, and the sooner the better, because retrofitting the activity/workflow split into mature code is significantly worse than starting with it. The notebook is the right idea, hand-rolling it is a bad trade, and the mature workflow orchestration tools are all substantially better at it than a first attempt will be.

For a job that finishes in three seconds behind an idempotent handler, no. Retry it. A durable log of a process that costs nothing to repeat is bookkeeping you pay for and never read.

The interesting cases are in between, and they usually turn on a question the engines cannot answer for you: how much of this program is allowed to become the engine’s shape? A team that answers “all of it” gets a great deal out of orchestration. A team that answers “the outer loop only” often finds that the outer loop was not where the failure was.