Writing workflows

The authoring contract: what a workflow on this engine is, what the engine guarantees, what a workflow must supply in return, and how to test and draw one. A human or an agent should be able to write the next workflow from this page alone. The worked example at the end is example_test.go, compiled and run by go test, so it cannot drift from the engine.

Vocabulary

Term Meaning In code
purpose what a workflow achieves; its workflows are the ways of achieving it, one purpose per workflow Purpose, on Definition
workflow a kind of work the engine can run, declared once in code Definition, registered in Definitions
revision a workflow's step list as the code had it for an attempt; the same names in the same order are the same revision Revision, recorded on RunRecord
instance one run of a workflow, identified within its type by its key RunRecord; Key for keyed workflows, BusinessDate for date-keyed
step one unit recorded against an instance: a stage, or one account's end of day StepResult
stage a step of a pipeline as code: a name, a Run, an Until or both, an Undo when its work must not outlive a failed run, and Final when its work must never be redone Stage
attempt one call of the runner for an instance; a retry is a new attempt at the same instance, never a new instance StartedAt is the attempt's start
child an instance started under another, its parent; the parent's children are the instances its sub-process started RunRecord.ParentID; RunFilter.ParentID lists them, Store.CountDescendants counts them by status at any depth; Store.DeleteRun removes a run and every run under it, for a host deleting the thing the run was about
sub-process step a step that starts instances of a child definition: one child, or children one at a time or all at once Step.Child, Step.Mode; the stage calls Runner.StartChild
mode how a sub-process step starts its children: one at a time is a loop, the next starting when the previous is through or when an operator starts it; all at once is a fan-out Mode: OneAtATime, AllAtOnce; unset means one child
given what a child is given, as prose: the engine hands it only its key and business date, so this is the contract of what the key names Step.Given
join the step whose Until waits for a child-starting step's children to be through Step.Joins, naming the child-starting step

An instance renders from the revision it recorded, not from the live definition, so it stays right after the code changes: Definition.StepsFor is its step list, described by the definition where a name still exists.

Declaring a workflow

A definition is the workflow as the registry, the pages and the diagrams see it. Declare it next to the code that runs it and append it to Definitions in an init, as gobank-deploy does. The registry starts empty, so a host's pages show only the workflows it runs: the library declares DailyUpdate beside RunDaily and leaves registering it to gobank; the demo app registers that, its own release pipeline, and a payment batch with its payments as children.

Field What it is Rule
Type the workflow's name lower snake case; unique across every program sharing a store
Purpose what the workflow achieves: a Name and a Description share one Purpose value between the workflows that are ways of achieving the same thing
Description what one instance achieves and what a retry does one or two sentences
InstanceKey what identifies one instance, as prose "version", "environment@expiry", "business date"
Steps every step the runner will record, in order Name matches the stage's Name exactly; Description says what it does; Source is an https:// link to the code
Steps[i].Child, Mode, Given the step is a sub-process: it starts instances of the Child definition, one, OneAtATime or AllAtOnce, each Given what the prose says the child is a definition of its own, with its own page and picture
Steps[i].Joins the step waits for every child of the named step to be through name the child-starting step; a fan-out nobody joins is warned about on the page

Check the registry in the host's tests, as the demo's definitions_test.go does: every definition has a type, a purpose and a description, every step links to its code. The instance key is the one idempotency rule, so choose it from the system under control, not from the clock: a demo environment is keyed by its expiry, which the server carries, so a restart finds the same instance.

Stages

A pipeline is Runner.RunPipeline(ctx, type, key, date, stages). Each stage runs in order and is recorded as a step with the stage's name. A stage has a Run, an Until, or both: Until is a condition the stage waits on, and Run executes once it holds. Undo reverses the stage's work when a later stage fails.

Until Run Step recorded Run status RunPipeline returns
unset returns nil completed continues
unset returns error failed, with the error failed nil, the error
false not called waiting, once, when the wait began waiting the run, nil
true unset completed continues
true returns nil completed continues
true returns error failed failed nil, the error
error not called failed, with the error failed nil, the error
completed in an earlier attempt not called kept continues
after the last stage completed the run, nil
key already completed not called nil, nil

When a stage fails, the completed stages before it are undone in reverse order, nearest first, each by its own Undo; a stage without one keeps its step and its work.

Completed stage before the failure Undo Step The undoing
has no Undo not called kept passes over it
Undo returns nil called removed continues to the one before
Undo returns error called kept, its work may still stand stops; the error is reported with the failure

The run's error names the failed stage, then what was undone and any undo that failed: days: the demo stopped answering (undone: add, create); undo add: delete server: forbidden. Give a stage an Undo when its work bills or blocks while the run sits failed: a server created, a lock taken. A tag pushed or a figure recorded is left to stand.

A waiting run is parked, not held: RunPipeline returns, and the host calls it again on its own tick. The cadence belongs to the host, so there is no interval on the stage. An approval, a scheme response or a server coming up is a store write or an observation whose condition is "is it there yet". Do not sleep and poll inside a stage; put the condition in Until and let the tick do the polling, so the wait survives a restart.

Resume

A failed instance is retried by calling the runner again with the same key: the failed step is cleared, the run restarts, and the stages run again from the first one not completed. A waiting instance is re-entered the same way, but it continues the same attempt: nothing is cleared and StartedAt is kept.

Call for an existing instance Steps cleared Run Stages run
after a failure or a crash (failed, running, pending) failed, running, pending; the undone ones went when they were undone RestartRun: new attempt, StartedAt now, the stages' revision recorded the first not completed, onwards
while waiting none ResumeRun once a condition holds, same attempt the waiting one, onwards
after completion or cancellation none; the call is a no-op

This is what resume demands of a stage:

  • Idempotent. A stage may run again after a crash that lost its step. Make its effect safe to repeat: create only when absent, push only when not already pushed.
  • State on the system under control, not in the run. The run records that a stage completed, nothing more. What the stage made (a server, a tag, a figure) lives where it belongs and is found again from there.
  • Resumable from any stage. A later stage cannot assume an earlier one ran in this process. Anything it needs, it reads back.
  • Undone when it must not outlive a failure. A stage whose work bills or blocks while the run sits failed has an Undo that removes it; the next attempt then makes it again. Its Undo is as idempotent as its Run: deleting what is already gone is not an error.
  • Its own record, if it has one. A workflow with domain figures to keep (a drill's readings, a performance run's rates) keeps them in its own table against the run's id, found through FindRunByKey.

Rewind

An operator can send an instance back to an earlier stage with Runner.Rewind(ctx, type, key, from, stages): when a check finds a problem that an earlier step must fix. The stage named from and every later one lose their steps, and the run starts a new attempt; the host's next call to RunPipeline runs them again, and the stages before from are not repeated. Rewind does not run anything itself.

Instance from Rewind Next RunPipeline
any, but no runner on it at the moment (a crash's running is fine) a stage, no Final stage completed at or after it steps from from on removed; run running, new attempt runs from from
any a stage at or before a completed Final stage refused unchanged
any not a stage of the pipeline refused unchanged
none for the key — refused —

Mark a stage Final when its work must never be redone, such as an archive written or money moved. Rewinding after it is allowed. The steps a rewind removes are not kept as history yet (#5).

Sub-processes

A workflow whose instances have parts with lives of their own — a batch of loads, a load of photos — is three definitions, not one long one. The parent's sub-process stage starts each child with Runner.StartChild and returns; the child is a pending instance of its own definition, which the host's tick runs by its key like any other. The stage reads back its own run id with FindRunByKey, as it reads back anything else it needs.

Sub-process stage's Run calls StartChild The child key StartChild Then
for a new child not in the store creates it: pending, ParentID the parent's, no attempt yet the tick's RunPipeline(child type, key, …) finds it by key and runs it, keeping the parent link
again, after a crash or a retry in the store, any status returns it as it is the tick carries on with it
under a run id the store does not have — error the stage fails

A stage that starts children all at once never waits for them: a child whose stage waits on its parent's later stage would deadlock against a parent waiting on the child. Waiting is a later stage's Until, through the existing park-and-poll: CountDescendants by status when the rule is "every child done", or the host's own condition when it is something of the domain, such as "every photo receipted". That stage names the child-starting step in Joins, so the definition page and the picture show where the children are waited for. A stage that starts them one at a time is a loop, and is its own join unless a later stage is named. The instance page lists a run's children with their status, and a child's page links up to its parent. The demo's payment batch (cmd/demo/payments.go) is the pattern end to end: start payments starts one child per payment, settle joins them, waiting until none is active and failing the batch if any failed, and one press of Pay is the host's tick run until the batch settles.

The engine hands a child only its key and business date (StartChild): the key names the host's own record, which the child reads. Given on the child-starting step says what that is, as prose, and the child's definition page repeats it under Started by.

Mode Joins names the step Drawn as The definition page says
unset, no Child — a box —
OneAtATime no a loop arrow back to the step starts children one at a time and waits for each: the step is its own join
OneAtATime yes a loop arrow, a dashed edge into the joining step starts children one at a time; step waits for all
AllAtOnce yes stacked boxes, a dashed edge into the joining step starts a child per item, all at once; step waits for all
AllAtOnce no stacked boxes the same, with a warning: nothing waits for the children

What the store records and what the host supplies

The store records runs and steps: workflow_runs and workflow_steps, versioned through sqlstore.Migrations. Use sqlstore.New on a pglike database for real, NewMemStore in tests. A step record is the fact that the step ran: its name, status, error and duration. It is not what the step worked on.

Step data lives in the host's tables

The engine never sees a stage's inputs, so it cannot record them, and it should not: a posting instruction is bank data, and bank data belongs in a typed table under the bank's schema, not in a payload column on the engine's. The pattern is the host's own table, in the same database, keyed to the engine's rows by run_id (what an instance is about) or step_id (what one step did). A restart with only the database then finds both where to resume and what to do.

Data Lives in Keyed by Demo
that a step ran, and how it ended workflow_steps post failed in 1.07ms: posting PAY-0001/3 failed
what an instance is about the host's table run_id payments: account, amount, posted_at
what one step produced the host's table step_id an account's day: balances, interest
the state the steps change the host's table, keyed by its own id accounts: the balance now; the day's steps and the payments move it
what to do next the runs (FindRunByKey, the last completed date) key the batch to retry, the date to process
anything process memory nothing: the simulation's counters are the remaining exception

Write the row in the stage that first holds the data, and make the write idempotent on its natural key, since a stage may run again after a crash: the demo's receive inserts the batch file with ON CONFLICT DO NOTHING, start payments keys each row to the child run it starts, post stamps the row posted. Every later stage and the instance page read the row back rather than carrying a struct from memory: if the struct is the source, the database is not. The host owns the table's migrations beside the engine's, in one sequence in gobank, as its own DDL at open in the demo.

The daily step's balance and interest columns on workflow_steps predate this pattern. They are the exception, not the model.

The host supplies:

What Why
a store sqlstore.New(db) or NewMemStore() the engine keeps no state of its own
its step data tables keyed by run_id or step_id, as above the engine records that a step ran, not what it worked on
the tick a loop that calls the runner until the run is not waiting the engine never holds a goroutine
a log per step an io.Writer the stages write to the step record keeps only name, status, error and duration
the pages instance and definition pages; the demo app and gobank-deploy are two. view.Views.Progress is a run's steps as a row of coloured dots for the host's own page about the thing the run is about; the instance page carries the same row under its title the engine renders nothing
the clock, injectable Now and Sleep as fields with real defaults tests drive time

Testing a workflow

Drive one key against NewMemStore() with fakes for the system under control and an injected clock, then read the store back. runnertest has the two checks every workflow test makes: OnlyRun is the single instance, Steps is each step's status by name. A failure test arms one fake to fail, checks the run is failed with the failed step, then calls again and checks the same instance completes with the earlier stages not repeated. A waiting test flips the condition between calls.

A new store implementation runs storetest's contract checks, as sqlstore and MemStore do.

Drawing a workflow

diagram.Definition writes one d2 file per definition from the registry (the demo's defd2 does, and task docs:d2 renders them): boxes are the steps, each linked to its source. A sub-process step is one collapsed box, BPMN's ⊞ on its name, a loop arrow when it starts children one at a time, stacked when all at once, linking to the child definition's page; the child's picture is on that page, and a dashed edge runs from the step to the one that joins its children. Each definition is drawn twice, across in one row (diagram.Across, <type>.svg) and down in one column (diagram.Down, <type>.down.svg); the pages show the one that fits the screen, down on a phone. Neither is a grid: d2 fills a grid column first and cannot loop an edge back to a grid cell, and in a row or a column no edge crosses a step. A long row shrinks to fit the page, which is the pressure to split: one picture answers one question, so a definition that does not fit across a screen becomes a sub-process with its own page for the run of steps that overflows. diagram.Instance draws one instance with its steps coloured by state; the demo app serves it drawn across at workflows/instances/<id>/diagram.d2. Nothing is drawn by hand, so the pictures cannot drift from the code.

Worked example

A promotion: build, wait for an approval, promote. From example_test.go.

func promotion(approved *bool, log func(string)) []wf.Stage {
	return []wf.Stage{
		{Name: "build", Run: func(context.Context) error { log("building"); return nil }},
		{Name: "approve", Until: func(context.Context) (bool, error) { return *approved, nil }},
		{Name: "promote", Run: func(context.Context) error { log("promoting"); return nil }},
	}
}

func ExampleRunner_RunPipeline() {
	ctx := context.Background()
	store := wf.NewMemStore()
	approved := false
	stages := promotion(&approved, func(s string) { fmt.Println(s) })
	runner := wf.NewRunner(store, nil)
	date := wf.NewBusinessDate(2026, time.October, 8)
	for tick := 1; ; tick++ {
		rec, err := runner.RunPipeline(ctx, "promotion", "v0.3.0", date, stages)
		if err != nil {
			fmt.Println("failed:", err)
			return
		}
		fmt.Printf("tick %d: %s\n", tick, rec.Status)
		if rec.Status != wf.StatusWaiting {
			break
		}
		approved = tick == 2 // the approval arrives between the second and third ticks
	}
	// Output:
	// building
	// tick 1: waiting
	// tick 2: waiting
	// promoting
	// tick 3: completed
}

The test beside it, TestPromotionRecordsEveryStep, is the testing pattern above: one key, the store read back through runnertest.

Checklist

  • A Definition with type, description, instance key and every step named, described and linked to its code, appended to Definitions
  • The instance key comes from the system under control
  • Each stage is idempotent and reads back what it needs
  • Waits are Until conditions, not sleeps
  • A stage whose work bills or blocks has an Undo
  • A stage whose work must never be redone is Final
  • A sub-process stage starts its children with StartChild in its Mode, saying what they are Given; a later stage's Until waits on them and Joins it
  • Each definition fits across one screen; a longer run of steps is a sub-process with its own definition and page
  • Now and Sleep are injectable with real defaults
  • A test for the happy path, one for a failure resumed, one per waiting stage, one for a failure undone where a stage has an Undo
  • Domain figures in the workflow's own table against the run id
  • The host ticks waiting runs and shows the pages