Poiesis v0.2: one task, one Kubernetes Pod

Poiesis v0.1 ran every task as a small fleet: an orchestrator Pod, three worker Jobs, one more Job per executor, and Redis in the middle holding it all together, so a task with three executors started seven Pods. In v0.2, the same task is one Pod.

It turns out Kubernetes already had the feature I’d spent months building by hand, and I simply hadn’t noticed. This post is about how the first design met reality, what it cost to run, and what got deleted.

What a TES task is

Poiesis implements the GA4GH Task Execution Service (TES). If you haven’t met TES before, the v0.1 post has the long version, but the short one is enough for what follows:

• A Task is an ordered list of Executors, each a container with a command, env and some inputs on disk.
• Executors run strictly in order. If one fails, the rest never run.
• Inputs are staged in from S3 or HTTP before the first Executor, and outputs are uploaded after the last.
• Every Task ends as COMPLETE, EXECUTOR_ERROR, SYSTEM_ERROR or CANCELED.

That list comes back later, so it’s worth keeping in mind.

v0.1: the obvious design

When I started, the obvious design was a small fleet where each component owned one part of a task’s life. Anyone who has been near a batch system knows the shape: an orchestrator, some workers, a message broker and a database.

The v0.1 fleet:

• API server. connexion, speaking GA4GH TES over REST. Stateless, scales horizontally, writes Task records to MongoDB.
• Torc (Task Orchestrator). One long-lived Pod per task. Creates the PVC, launches the three workers in order, cleans up.
• TIF (Task Input Filer). A Job that stages inputs from S3 or HTTP onto the task’s PVC.
• Texam (Task Executor and Monitor). A Job that itself launches one Job per executor, in order, and watches each one.
• TOF (Task Output Filer). A Job that uploads outputs from the PVC once every executor is done.
• MongoDB. Tasks, executors, logs.
• Redis. Pub/sub. Each worker publishes “done” on a channel keyed by task ID, and Torc blocks until it hears the messages it expects.

The flow went like this: the API writes to Mongo and starts a Torc, which creates a PVC, launches TIF and blocks on Redis until TIF reports that it’s done. Torc then launches Texam, which starts the executor Jobs one at a time, watches each of them and reports back in turn, after which Torc launches TOF and waits once more. When TOF reports done, Torc writes the final state to Mongo and exits.

It worked well enough to run real tasks, and along the way it grew a Helm chart, OIDC auth, per-task Kubernetes config, glob-style output paths, multi-arch images and a Nextflow guide. Tasks went in and results came out, and the whole time I kept paying a tax on it.

What v0.1 cost

The tax came in three forms, mostly.

1. Every task cost 4 + N Pods. A task with three executors started seven: Torc, TIF, Texam, three executors and TOF, and each of them paid its own scheduling delay, image pull and container start. For a workflow engine sending hundreds of small tasks, that’s the difference between snappy and go-make-a-coffee, and for the operator it meant the scheduler and admission quotas were busy with my orchestration instead of anyone’s actual work.

2. Redis was on the critical path. Every task needed Redis messages to move from one step to the next, so if Redis hiccuped, tasks froze. Redis is fine software, but it’s stateful software, and putting stateful software in the hot path of a thing whose whole job is scheduling other software is asking for trouble. Operators also had to run a Redis they understood, monitor it, back it up and add it to their compliance scope, and nobody installs a TES because they were also hoping to install a Redis.

3. When Torc died, nobody noticed. Torc was the only thing that knew a task was in flight, so if it died after launching Texam but before hearing back, the children kept running, finished, published into the void, and the task sat in RUNNING in Mongo forever. There was no reconciler, and the “monitor” Job I wrote to paper over the worst cases didn’t change the fact that the hole was in the design.

There were smaller cuts too: MongoDB couldn’t enforce the TES schema, so the shape of stored documents slowly drifted; auth was bolted on; cancelling took effect “when the next pub/sub message arrived”, which was sometimes never; and Kueue was off the table, because Kueue admits Jobs and a Poiesis task was a whole tree of them.

None of this made v0.1 broken, but it did make it expensive to run, and over the life of a project those two end up being the same thing.

The realisation

I’d accepted 4 + N Pods as the cost of doing business, because how else do you run N containers in order, with inputs before and outputs after? Something has to launch them, something has to watch them and something has to coordinate, and I’d assumed that was simply what orchestration looks like.

Kubernetes, it turns out, already has a primitive for “run these containers one at a time, stop at the first failure, and let them share a filesystem”. It’s called init containers, and it had been sitting there the whole time.

Put it next to the TES list from earlier:

• TES executors run in order. Init containers run in order.
• TES stops at the first failed executor. Init containers stop at the first failure.
• TES executors share a working directory. Init containers in one Pod share volumes.
• TES inputs come before executors and outputs after. Init containers run in the order you declare them.

The match wasn’t rough, it was exact: I had written a distributed orchestration layer to do, slowly and over Redis, what one Pod spec does on its own.

I had actually considered this for v0.1 and talked myself out of it, which is recorded for posterity in the v0.1 post. The problem was the watcher: something inside the Pod has to watch the init containers and write each state change to the database, because the API server outside can’t see init-container progress in real time, and a normal sidecar doesn’t help because it only starts after the init containers finish, by which point the executors are done.

What changed is that Kubernetes 1.29 shipped native sidecars: init containers with restartPolicy: Always. A native sidecar starts before the init containers that follow it, stays up for the whole life of the Pod, and is shut down after the main containers exit, which is exactly the shape of “something that watches the executors and writes down what happened”. 1.29 is old news by now too, since any cluster a TES deployment realistically targets has been past it for a long time, and once native sidecars existed there wasn’t much left to design.

v0.2: the TaskPod

In v0.2, one task is one Kubernetes Pod, wrapped in a Job. In order, the Pod runs:

• TRec (Task Recorder). The native sidecar, which starts first, stays up for the life of the Pod, watches its own Pod through the Kubernetes API, and writes state changes and per-executor logs to Postgres as the other containers progress.
• TIF, as an init container, staging inputs onto the task’s PVC.
• N executor init containers, one per TES executor, kept in order by Kubernetes itself.
• TOF, as an init container, uploading outputs.
• ack, the main container. Kubernetes insists that a Pod has at least one regular container, even when all the real work happens in init containers, so the last thing in every task is a tiny step that reuses the Poiesis image, exits 0 and moves the Pod to Succeeded. It makes me happier than it should that every task ends with a container whose only job is to say “ok”.

The whole manifest comes out of one pure function in jaeaeich/poiesispoiesis/core/taskpod.py↗. Stripped down, it looks like this:

kind: Job
spec:
  template:
    metadata:
      labels:
        poiesis.io/task: <task id>
    spec:
      restartPolicy: Never
      initContainers:
        - name: trec          # native sidecar, runs throughout
          restartPolicy: Always
        - name: tif           # only if there are inputs
        - name: exec-0
        - name: exec-1
        - name: exec-2
        - name: tof           # only if there are outputs
      containers:
        - name: ack           # exits 0: Pod Succeeded

All of them share one PVC, sized from the task’s disk_gb, whose ownerReferences point at the Job so that Kubernetes deletes the volume when it deletes the Job. There’s no cleanup code of my own anywhere; if Kubernetes is up, cleanup happens.

Outside the TaskPod, one global piece is left:

• TCtl (Task Controller). A small Deployment with 2 or 3 replicas and leader election through coordination.k8s.io/leases. It runs an informer on Pods labelled poiesis.io/task, and if a TaskPod ends and Postgres has no final state for it, TCtl writes one, including the reason Kubernetes gave (OOMKilled, Evicted, Error).

TCtl is not on the happy path, since normally the TRec writes the final state itself. It exists for the cases where the TRec can’t, because the node died, the TRec itself got OOM-killed, or the Pod was evicted before it could finish. Argo Workflows and Tekton have the same shape, with an agent inside the Pod doing the work and a controller outside for when the agent can’t speak for itself.

That leaves one Pod per task, three components (the API, the TaskPod and TCtl) and two dependencies (Kubernetes and Postgres), which is the entire system.

What got deleted

Three components left the codebase in the redesign:

• Torc. Its only job was starting things in order and waiting on Redis, and init containers do both for free, with stronger guarantees.
• Texam. Its only job was starting one Job per executor and watching it, which is the same story; an executor is now just an entry in the TaskPod’s list of init containers.
• Redis. With nothing left to coordinate across Pods, Redis had nothing to do, so it’s gone from the chart, the docs, the security review and the operator’s head.

After the rewrite I ran vulture, a Python dead-code detector, over the repo, and it had a very good day.

Postgres, and the document store finally biting

The other half of v0.2 was moving off MongoDB, which was fine as far as it went: it ran, and it stored documents. The trouble was that it had no opinion about what those documents looked like, whereas the TES schema is precise, with specific fields on tasks, ordered executors and logs that nest in specific ways. In v0.1 nothing enforced any of that at the storage layer; Pydantic checked it on the way in and nothing checked it after, so small drift piled up. If an auditor had asked me what the source of truth for the schema was, the honest answer would have been “the application code, which several people have refactored several times”, and you can’t give that answer in a regulated environment, when being able to hand the system to someone in a regulated environment is the whole point of GA4GH compliance.

v0.2 runs on Postgres with a proper relational schema: foreign keys, NOT NULL constraints, enum types for final states, and separate task_log, executor_log and system_log tables that match the spec one to one. Migrations are real migrations rather than “the code will probably write the new shape eventually”, and concurrent writes are safe because the database handles concurrency instead of a layer of hopeful application logic.

It was unglamorous work that I’ll be quietly grateful for every time someone asks me for a schema diagram.

Small decisions that mattered

The TaskPod can read Pods and do nothing else. The TRec needs to watch its own Pod, so the TaskPod’s service account can get, list and watch Pods in the task namespace, with no writes and nothing cluster-wide, which means a user’s task can’t use those credentials to change anything.

No heartbeats. An earlier sketch had the TRec sending heartbeats and a sweeper killing tasks that went quiet, which was wrong, because a sweeper that only looks at the database can’t know why a task died. OOMKilled, Evicted and Error mean different things to the person who submitted the task, and only the Kubernetes API knows which one happened, so TCtl asks Kubernetes directly and writes the real reason to Postgres. The happy path never touches it, and the unhappy path gets a better answer than a heartbeat could have given.

Two Kubernetes settings instead of a janitor. ttlSecondsAfterFinished on the Job and ownerReferences from the PVC to the Job mean cleanup is configured once in the manifest, with no reaper code, no janitor cronjob, and no “the app deleted it, but is the PVC still there?” bugs, because Kubernetes owns it.

FastAPI replaced connexion. A boring rewrite with no performance story: connexion’s spec-first approach was costing more friction than it saved, and FastAPI’s Pydantic-native style matched how the rest of the code was already written. Sometimes a refactor is just paying back an early framework choice.

What v0.2 buys

If you run Poiesis, you now run Kubernetes and Postgres and nothing else stateful. Each task is one Pod instead of 4 + N, and every piece of it carries a poiesis.io/task=<id> label, so one selector finds all of it. The chart declares that it needs Kubernetes 1.29, so an install on an older cluster fails straight away instead of halfway through.

If you submit tasks, they start faster, because there’s one Pod to schedule instead of seven, and GET /tasks/{id} is accurate within seconds because the TRec writes as each container finishes. CANCELED actually means cancelled, even in the middle of an executor, and SYSTEM_ERROR comes with the real reason from Kubernetes instead of a guess.

As for me, I have much less code, a schema I can point at, and tests I can reason about, because there’s no Redis to mock and the whole orchestration model is “Kubernetes ran this Pod”.

What I’d tell past me

It would have been easy to keep adding features to v0.1, because it worked, the fixes kept landing, and the people running it weren’t complaining loudly. They were quietly putting up with the operational tax, because TES on Kubernetes isn’t a crowded space and the alternatives weren’t obviously better.

The better move was to stop and ask: if I started today, knowing what Kubernetes gives me now, would I build it this way? The answer was no, by a wide enough margin that patching didn’t make sense.

Most rewrites are mistakes, and the ones that aren’t usually happen because the platform underneath you grew a feature that does one of your components’ jobs. Init containers were always there, native sidecars closed the last gap, and once both existed v0.1 wasn’t engineering anymore so much as carrying around the old way of doing things.

The code is at jaeaeich/poiesis↗ under Apache 2.0, and the Helm chart is in deployment/helm. If you run TES on Kubernetes, or you have opinions about how you would, open an issue, because I want to hear about the cases I haven’t seen yet.