I built a TES server because I wanted to understand one

In the summer of 2024 I worked on TESK, the reference Kubernetes implementation of the GA4GH Task Execution Service, as a Google Summer of Code contributor with ELIXIR Cloud. I learned a lot from that codebase, and I also came out of it with a pile of opinions about what I would do differently if I started over.

There is only one way to find out whether opinions like that are any good, which is to actually start over, so that is what I did. Poiesis is a TES server written in Python and built for Kubernetes from the first commit, and v0.1.0 went out this week. This post covers what TES is, why anyone should care, the design I ended up with, and the parts of that design I already know are going to bite me.

What is TES, and why bother

If you have never run a bioinformatics workflow, feel free to skip this section, and if you have, you already know the pain this is about.

A research group runs an aligner, a variant caller, three QC steps and an annotator. Each step is a container, and a workflow engine (Nextflow, Snakemake, Cromwell, take your pick) wires them together. The whole thing then has to run on whatever the institution has: a Slurm queue, some HPC scheduler, a Kubernetes cluster in a cloud, sometimes all three depending on which grant paid for what.

The engine and the cluster don’t speak the same language, because the engine thinks in tasks while the cluster thinks in jobs, pods and Slurm scripts. Every engine therefore grew its own adaptor for every cluster manager, and the world ended up with N×M adaptors that nobody wants to maintain.

GA4GH (the Global Alliance for Genomics and Health) fixed this by putting one REST API in the middle: the engine submits tasks to a TES server, and the TES server runs them on whatever cluster it sits on. Write one TES server per kind of cluster, and any engine that speaks TES can use any of them.

The contract is small and precise:

• A Task is an ordered list of Executors.
• An Executor means “run this image with this command and env, with these inputs on disk, and capture stdout and stderr.”
• Executors run strictly in order, so Executor 2 doesn’t start until Executor 1 has succeeded, and if Executor 1 fails the rest never run.
• Inputs are staged in from object storage (S3, HTTP) before the first Executor, and outputs are uploaded after the last.
• Every Task ends in one of four states: COMPLETE, EXECUTOR_ERROR, SYSTEM_ERROR or CANCELED.

That is the whole contract: strict ordering, a shared filesystem, inputs first, outputs last, four ways to end. The catch is that you’re running all of it on Kubernetes, which has its own ideas about containers and lifecycles that don’t quite line up with TES, and that gap is where the actual engineering happens.

Why not just use TESK

You can, and if you have a TES-shaped problem today and need something that runs, you probably should, because TESK works and runs in production at several institutions.

While I was inside the codebase, though, I kept noticing things I’d do differently: the data model carried history from earlier versions, the Kubernetes side played it safe in places where newer primitives could have helped, and I wanted to design the operator side (Helm chart, secrets, RBAC) from a blank page instead of evolving it.

None of that is a criticism of TESK so much as the itch you get after months inside any codebase. You can scratch it by sending patches, or by building the thing you wish existed and bringing what you learn back, and I picked building because I wanted to see the design space without having to stay compatible with anything.

Poiesis is Greek for “the act of bringing something into being”, which felt right for a thing whose job is bringing tasks into being on a cluster. It was also free on PyPI, which is half the battle with naming software.

The architecture

A TES server on Kubernetes has to answer, concretely:

• Where does a task live while it runs?
• How do executors share a filesystem?
• What enforces the ordering?
• Who watches the executors and writes their state down?
• How does the API server know what’s happening inside a running task?
• What happens when any of the above dies?

Here are v0.1’s answers, and the short version is that a task runs as a small fleet of Kubernetes Jobs, coordinated by a long-lived orchestrator.

When the API receives a CreateTask, it writes a Task record to MongoDB, creates a PVC sized to the requested disk, and starts a Pod called Torc (Task Orchestrator), which owns the task from then on.

Torc launches three workers, one after another:

• TIF (Task Input Filer). A Job that mounts the task’s PVC and stages every input onto it, pulling from S3, HTTP or the inline content field depending on the URL scheme.
• Texam (Task Executor and Monitor). A Job whose only purpose is to launch and watch your executors. It creates one Kubernetes Job per executor, in order, all mounting the same PVC, waits for each to finish before starting the next, and stops at the first non-zero exit.
• TOF (Task Output Filer). TIF in reverse: it walks the output list, expands globs and uploads each file to wherever its URL points. When it’s done, Torc writes the final state to Mongo and exits, and Kubernetes cleans up the rest.

1. TorcorchestratorPod
2. TIFPod
3. TexamPod
4. exec 1Pod
5. exec 2Pod
6. exec 3Pod
7. TOFPod

Redis pub/sub: each step reports back to Torc

Torc and its children talk over Redis pub/sub: each child publishes a “done” message on a channel keyed by the task ID, and Torc blocks until it has heard from everyone it’s waiting on. I picked Redis because it’s cheap to run and does this “collect messages from N workers” pattern out of the box.

The names are on purpose. Torc, TIF, Texam and TOF show up in logs, in Pod names and in metric labels, so once you know those four words the whole system is easy to talk about, and once a team is running something, being able to talk about it matters more than the code.

The parts I’m proud of

Sequencing lives in Torc, not in the Pod spec. I considered putting inputs, executors and outputs into one big Pod with N init containers, but the number of executors comes from the user, and there are plenty of corners (cleanup, restarts, knowing exactly which executor failed) where having each phase as its own Kubernetes object gives me cleaner edges to reason about. Separate Jobs cost Pod startup time, and in return I get clear phase boundaries and a failure story for each phase, which for tasks that usually run for minutes or hours is a trade I’m happy to make.

Pydantic end to end. Every request, every internal model and every stored object goes through Pydantic, with the TES schema as the source of truth, generated into code where possible, so the API rejects any task that doesn’t validate. This has caught more bugs than I’d like to admit.

OIDC first. Poiesis is a plain OIDC resource server, so it works with Keycloak, with institutional identity providers and with the big clouds. There is a dummy auth mode for local development, which complains loudly in the logs so nobody ships it by accident, and there is no username and password story at all, because identity is somebody else’s job and the institutions this is built for already have an IdP.

Helm chart from day one. The chart is the supported way to install Poiesis, with bare manifests only for development, and committing to that early made me treat config, secrets, RBAC and resource limits as design decisions instead of afterthoughts.

Pluggable filers. Inputs and outputs go through a FilerStrategy interface, with S3Filer, HttpFiler and ContentFiler in v0.1. A new storage backend is one file, so the first person who shows up wanting GCS has an obvious place to put it.

Globs in outputs. Real bioinformatics tasks tend to produce “everything under /work/results/” rather than a tidy list of paths, so outputs accept glob patterns and expand them at upload time. It sounds minor until you try a TES server without it, at which point it turns out to be the whole ballgame.

Per-task Kubernetes config. Resources, node selectors, tolerations and security contexts can be set per task through backend_parameters, with operators setting defaults and limits in the chart and users overriding within them. It sounds like a yak-shave right up until someone needs a GPU executor on a tainted node, and from that moment on it’s the only thing that matters.

The parts I know will bite me

I’m not naive about this design, and some of it is going to need work.

Every task costs 4 + N Pods. Torc, TIF, Texam and TOF, plus one per executor, so five executors means nine Pods to schedule, and clusters with tight admission quotas will feel it. I don’t have a clean fix yet, because the obvious alternative, one Pod with init containers, has its own problem: something still has to watch the executors, and it’s not clear where that watcher would live.

Redis is stateful, and it’s on the critical path. If Redis hiccups, tasks freeze, and operators have to run a Redis they understand and add it to their compliance scope. I’m accepting that because the alternatives (a custom controller, or heartbeats into the database) would have been a lot more code for a v0.1, but it’s a real cost, and sooner or later a serious operator is going to ask me to get rid of it.

MongoDB can’t enforce the TES schema. Pydantic checks it on the way in, but nothing checks it on the way out, so if the models drift, the stored documents quietly drift with them. “Trust the application code” is a fine audit story for a v0.1 and a much worse one for a regulated environment.

Cancelling waits for the next Redis message. Cancel a task in the middle of a long executor and it stops when that executor finishes rather than when you pressed the button. There is a mitigation in the code, but it’s “best effort”, which is polite English for “sometimes”.

No Kueue. Kueue admits Jobs, and a Poiesis task is “the Torc Pod and everything it creates”, which isn’t one Job, so operators who want Kueue-style batch admission can’t have it in v0.1. This one bothers me the most, because it’s baked into the structure and no patch will fix it.

I’m putting this list on the public internet partly because it’s true, and partly so that the next time I open this codebase I can read it and decide what to fix first.

What v0.1 is and isn’t

It is:

• Compliant with the GA4GH TES 1.1 schema.
• Tested as a Nextflow backend, on a real cluster, with a real pipeline.
• One helm install and a Mongo connection string away.
• Documented, with end-to-end guides for a local cluster backed by MinIO and for generic OIDC.
• Apache 2.0.

It isn’t:

• Battle-hardened. v0.1 means v0.1: it has run real tasks, but it hasn’t run a million.
• Built for tiny tasks. 4 + N Pods is what it is, so submit 10,000 one-second tasks a minute and you’re going to have a bad time, whereas 100 tasks a minute that each run for minutes or hours will be fine.
• Strictly multi-tenant. There’s one database and one task namespace, so if you need hard isolation between tenants, run one Poiesis per tenant.

The code is at jaeaeich/poiesis↗. The Helm chart is in deployment/helm, the docs are linked from the README, and the Nextflow guide is the quickest way to convince yourself it works.

If you run TES on Kubernetes, or you’re thinking about it, or you just have opinions about how it should work, please open an issue. I’ve spent enough months alone with this codebase that I’d really like to hear where my taste is wrong.