Optimizing a handrolled S3 client

S3, AWS’s object storage, is proprietary, but the wire protocol is just HTTP plus SigV4 plus a smattering of XML, and at some point I asked myself, in the cocky way only a Rust programmer can be cocky, how hard it could really be to build a high-performance S3 client from scratch.

The protocol turned out to be the easy part. Everything else (connection pools, multipart races, signing on retry, the syscall pattern your kernel actually likes) is where the bodies are buried. s3z is what came out the other side: a Rust S3 client and library with Python and Node bindings, benchmarked against mc, s5cmd and aws-cli. This post is every wall I hit on the way there, including the ones I built myself.

Starting boring on purpose

The first thing I wrote had no networking at all, just types, config, credentials and errors, because I wanted the shape of the API nailed down before a single byte hit a socket.

The errors were typed from day one, since anyhow is great in a CLI but people using a library want to match on what went wrong rather than read a string. Then came the HTTP layer: percent-encoding object keys (S3 is very particular about this), SigV4 signing, XML parsing, and a retry loop that treats 5xx and 429 as worth another go. None of it was surprising and all of it was tedious.

After that the plan was a transfer engine, a CLI so I could type s3z up ./dir and have it just work, bindings for Python and Node, and a proper benchmark. That last one mattered most to me, because every README says “blazing fast” and almost none of them prove it.

The transfer engine

A naive S3 uploader looks like this:

for file in files {
    client.put_object(file).await?;
}

That works, but against a remote bucket it is close to useless, because you pay a full round trip per file and you pay them one after another. The upload engine in s3z is instead a producer feeding a pool of consumers.

A scheduler plans the multipart upload up front, taking a file size in and producing part offsets and sizes, then pushes those parts into a bounded channel that a fixed pool of workers pulls from at the other end. The channel is bounded so the scheduler can’t race ahead of the network and eat all your memory, and the pool is fixed so there’s a hard ceiling on how many requests are in flight at once.

Every multipart upload is wrapped in an AbortGuard, so if anything panics, errors or gets dropped halfway, the guard tells S3 to abort the upload on the way down. Without that, S3 will happily keep charging you for the orphaned parts, and I’d rather not be that person.

That all sounds clean, and it mostly is, but it also broke in several creative ways before I could trust it.

Dropped parts and stale signatures

The first real bug showed up as soon as the benchmark pushed real load: uploads would “complete” with chunks missing, and S3 would answer with 400 InvalidPart.

The worker loop was polling the channel with try_recv, grabbing a job if there was one and otherwise going off to check for finished uploads. Under load the scheduler outran the workers, and the poll could catch the channel looking empty at exactly the wrong moment and move on, at which point parts got dropped on the floor. The fix was to stop polling and use tokio::select! to wait on “a job arrived” and “an upload finished” at the same time, so a worker only wakes up when one of them is actually ready. Simplified, the change looked like this:

loop {
    match rx.try_recv() {
        Ok(job) => spawn(job),
        Err(_) => poll_completions(),
    }
    tokio::select! {
        Some(job) = rx.recv() => spawn(job),
        Some(done) = tasks.join_next() => record(done),
    }
}

While I was in there I found a second bug. A SigV4 signature includes a timestamp, and my retry loop was resending the same signed request every time, so if the first attempt took six seconds and got a 503, the retry went out with a stale timestamp and S3 rejected it as RequestTimeTooSkewed, as did the retry after that. The fix was simply to sign again on every attempt, which is obvious in hindsight and is now the first thing I check in any retry loop I write.

The optimization that made everything slower

At some point I decided the client should open 256 connections the moment it was created, so the first wave of uploads wouldn’t stall on TLS handshakes. I was quite pleased with myself, made the constructor async to accommodate it, and moved on.

It added somewhere between 250ms and a full second to every single run, which for a CLI you might call a hundred times an hour is a tax you pay forever. Worse, it made every benchmark bimodal: half the runs warmed up cleanly, half got stuck behind a slow handshake, and the charts looked like a Rorschach test.

Meanwhile reqwest’s pool already grows on demand, at about 2ms per new connection, which is nothing next to an S3 round trip, so I ripped my own optimization out. The bimodal spikes vanished, throughput didn’t move, and the constructor went back to being sync.

This is the first of several “I was wrong” moments in this post, and I’d rather show them than hide them, since performance work is mostly a matter of being wrong in interesting ways.

1.5 GiB of RAM to upload some 40 MB files

Files under the multipart threshold (50 MiB at the time) took a simple path: read the whole file into memory and send it. With 32 workers against a remote endpoint, where each upload sits in flight for hundreds of milliseconds, that added up to a peak of 1,566 MiB, which is a lot of memory to spend on uploading some 40 MB files.

Streaming the file from disk in 256 KiB chunks, the same way the multipart path already did, brought that down to 85 MiB with no loss in throughput. Those uploads also switched to UNSIGNED-PAYLOAD, because the chunked SigV4 signing dance is genuinely annoying and skipping it is fine over HTTPS (which you are using, I hope).

Nagle, jitter and other things you forgot existed

One round of tuning fixed four separate problems, each of them its own small headache.

Workers were still sometimes sitting idle while finished uploads queued up, because they were waiting on the job channel and nothing else, and putting both sides into the same select! kept every slot busy. When lots of workers got throttled at once, they all backed off by the same amount, retried in lockstep and got throttled together again, which a bit of random jitter in the backoff fixed without needing a new dependency.

Nagle’s algorithm and delayed ACKs were conspiring to stall writes by about 40ms each while the kernel waited for more data to show up, and turning on tcp_nodelay took care of that. Fresh connections were also negotiating HTTP versions for no reason, since S3 speaks HTTP/1.1, so forcing HTTP/1.1 skips the negotiation entirely.

None of this is exotic, but you only find it by running the real workload and watching what the kernel does with it.

The tokio blocking pool is not your write thread

Downloads had their own version of this. Each part of a big download writes its chunks into the destination file with pwrite, and my first version sent every one of those writes through tokio’s spawn_blocking. That works, but for a 1 GB file it’s about ten thousand trips through a thread pool that everything else in your program is sharing too.

When the OS decided to flush its page cache under all that write pressure, everything piled up behind it, and the same 5 GB download on the same network took anywhere from 4 to 29 seconds.

The fix was to give each file one dedicated writer thread. The async download tasks send chunks to it over a bounded channel and it writes them out one after another, with no per-write scheduling and no fighting over the shared pool, so the same 5 GB download now takes 4 to 6 seconds every time.

Two smaller wins came out of the same area. Buffering 512 KiB before each write cut the number of syscalls from about 64,000 per GB to about 2,000, and I removed a cap on download part size that S3 never asked for, so a 10 GB file at concurrency 8 now comes down as eight 1.25 GB streams instead of forty 256 MB ones, which means fewer requests, fewer signatures and fewer round trips.

Part size should be a function, not a constant

The part size was hardcoded to 50 MiB, which is fine for some files and bad for most. A 50 MB file becomes a single part, so there’s nothing to parallelise, while a 10 GB file becomes 200 parts, which is a deep queue for 8 workers to chew through, and a 500 GB file would blow past S3’s limit of 10,000 parts. That last case now panics with a clear message, because quietly producing an invalid upload is not a user experience I’m willing to ship.

What I landed on is this: for uploads, aim for about twice as many parts as there are workers, because PUT latency varies and a bit of queue depth smooths it out, and for downloads, aim for one part per worker, because the server starts sending as soon as you ask and fewer Range requests win.

Then I noticed the concurrency itself shouldn’t be fixed either, since a 128 MB file with 8 streams just allocates 8 buffers and barely uses them. Small files now get 2 streams, medium ones 4 and big ones 8, which dropped memory for the small-file workload from 79–92 MB to 30–39 MB while big files still saturate the link.

The backend that wanted a Content-Length

I’d set up four S3-compatible backends locally (MinIO, RustFS, SeaweedFS and Garage) so I could catch my own protocol assumptions before users did, and RustFS, to my lasting amusement, rejected every single upload with UnexpectedContent. It took me longer than I’d like to admit to work out that it wanted an explicit Content-Length header on every signed request, even where reqwest would have filled one in later anyway.

It also caught a shortcut I’d taken with empty bodies, which I’d been treating as UNSIGNED-PAYLOAD when RustFS’s stricter validator wanted the real SHA-256 of zero bytes (e3b0c4..., a hash worth burning into your memory if you do anything near S3).¹ Once both were fixed, all four backends went green.

I’d love to tell you all four stayed in the benchmark, but RustFS started hanging at random under load, and after half a day of trying to pin down why I took it out, since stability is not something you can add by pinning an image version.

The benchmark was its own project

I did not expect to spend this much time on the benchmark, but the whole point of s3z is “this is faster”, so the benchmark has to be something I trust.

Every tool and every operation is a small plugin file, so adding s5cmd meant writing one file and touching nothing else. Every cell gets a warmup run, because cold starts make short benchmarks noisy, and noisy cells get more samples until the numbers settle while quiet ones finish early, so a quick run takes 3 to 5 minutes and a full one 10 to 15. There’s also a compare command that fails if a change made things worse, but only if the difference is statistically real (a Welch’s t-test) and bigger than the noise I’ve measured. That gave me a tight loop: save a baseline, change something, run, compare.

Even so, it almost lied to me once. mc, MinIO’s client, was losing badly against Garage, and I nearly published a chart that said so before working out that Garage is strict about regions and mc wasn’t picking up the region from the environment the way the other tools were. The bug was in my setup, not in mc.

Bindings, where the borrow checker meets FFI

PyO3 and NAPI-RS are both excellent, but they’re also both opinionated about what can cross into Python or JavaScript, and the rough edges are exactly where you’d expect them: anywhere a Rust type borrows something.

The clearest example was the list paginator, which I’d written like this:

pub struct ListPaginator<'a> {
    client: &'a S3Client,
    continuation_token: Option<String>,
    // ...
}

That’s perfectly nice Rust and completely unusable from Python or Node, because neither has any idea what that 'a means. The fix was to make the paginator own cheap copies of what it needs from the client instead of borrowing it, which costs a couple of Arc::clones and in return gives the same API in all three languages.

Node had one more surprise that cost me an afternoon. NAPI-RS already runs its own tokio runtime, so any code that tried to start another one and block on it would panic about nested runtimes, and the fix was a small helper that checks whether a runtime already exists and uses it if so.

npm also rejected the name s3z for being “too similar to existing packages”, so on npm it’s @jae_aeich/s3z, which means I got to name the thing twice.

The one I almost shipped

There’s one more bug I’m glad I caught, late one night while squinting at a benchmark cell that looked too good. If you asked the worker pool to run with zero workers, it returned an empty success: no error, no work done, all your items silently dropped. I’d written that early return “defensively”, and the same mistake let a multipart upload with zero concurrency send S3 a request with no parts in it.

Both now panic with a clear message, because if you pass zero workers you have a bug, and quietly doing nothing just hides it. That’s the only kind of defensive code I actually believe in, the kind that turns bad input into a loud failure rather than a quiet no-op.

If you’re keeping count, that’s the connection warmup, the polling loop, the reused signatures, the blocking-pool writes, the in-memory uploads and the fixed part size, all shipped and then un-shipped, plus one I haven’t mentioned: I’d turned on all ~80 of Clippy’s restriction lints at once, then spent 35 annotations suppressing warnings about perfectly normal Rust. It’s down to 8 lints I actually chose.

What the numbers say

The reference run is committed in the repo: three 256 MB files, 10 to 30 runs per tool, against MinIO, SeaweedFS and Garage running in Docker on my laptop. On downloads s3z is level with mc and ahead of s5cmd and aws-cli on all three backends, and on uploads it’s ahead of all three tools everywhere. The download numbers on MinIO are typical:

Speed is only half of it, though. The part I’m proudest of is how little memory it takes to get there:

The three fixes from this post that moved the numbers most:

None of this needed any cleverness, just running the benchmark, reading the chart and asking “why is this bimodal?” until it wasn’t.

Things I’d tell past me

• Write the benchmark before the third optimization. I wrote it after the fifth, and I’d have caught the warmup mistake much sooner.
• Bound every channel. If the producer is faster than the consumer, you can bound it, buffer it forever or drop it, and buffering forever isn’t really a choice, since it’s still dropping, just later and inside the memory allocator.
• Sign again on every retry. I’ll never write a retry loop again without first asking what the request depends on.
• Test against more than one backend. The Content-Length bug would have shipped without RustFS in the mix, and someone, somewhere, would have hit it on a backend I’d never tried.
• Permissive licenses or bust. s3z is MIT, and every dependency is permissively licensed. I checked.

It’s the friends we made along the way

The code is at jaeaeich/s3z↗. The CLI installs with one curl, the library is on crates.io, the Python wheels are on PyPI and the Node package is @jae_aeich/s3z on npm. There are also Axum, FastAPI and Elysia examples that all expose the same API, using s3z directly from Rust or through the bindings.

I learned more building s3z than I have building anything in a long time, mostly because every optimization I shipped immediately showed me something else that was wrong. If you’ve made it this far and want to break it in a creative new way, please do; issues are welcome.