Manas Mishra
Résumé

Case study Datavio · DevOps

Blue-green deploys that never interrupt a running workflow

Our backend ran long async workflows in-process, so every release had to wait for them to finish. I rebuilt deploys so any number of versions can run side by side — new traffic goes to the newest, and old versions finish what they started.

Role
Designed & built
Context
Datavio · FastAPI backend
Stack
Docker Compose · nginx · Bash · flock · cron
Runs on
Azure VM · uvicorn, 4 workers per release
Drain window for in-flight workflows (raised from 2 h for production)
3 h
Health gate before any traffic moves — 60 checks, 2 s apart
~2 min
Janitor sweep that reaps fully drained releases
15 min
Rollback — point nginx back at the .prev upstream
1 step
requests nginx a91c3e2 removed janitor · every 15 min 5d02f17 draining in-flight work up to 3 h e7b4a09 live · healthy passed 60 × 2 s gate .prev → one-step rollback
Three releases at once: the newest takes all new traffic, the previous one drains its in-flight workflows, and the oldest has already been reaped by the janitor.
TL;DR Deploying with docker compose up --build replaced the running containers, so every release had to wait for in-process workflows to finish. Now each release boots as its own Compose project named <sha>-<timestamp> on a fresh port, must pass a health gate, and nginx moves new traffic to it with a graceful reload. Older versions keep running until their in-flight work drains — up to 3 hours — and a cron janitor reaps them afterwards. Rollback is swapping a pointer.

01 · The problemDeploys had to wait for the work to stop

The backend is a FastAPI service on an Azure VM, and some requests kick off long-running async workflows inside the app process. Deploying was docker compose up --build — which rebuilds and replaces the running containers.

Anything still in flight would die with them. So every deploy started with a question nobody likes: is anything running right now? Releases became something you scheduled around the work, instead of something you just did.

Before

Wait until no workflow is running → compose up --build replaces containers in place → hope nothing started in between.

After

Boot the new version beside the old → health-gate it → switch new traffic → let the old one finish → reap it later.

02 · The intuitionStop replacing the thing doing the work

The fix isn't shorter workflows or a queue in front of everything. It's to stop replacing the process that's doing the work. If the old version keeps running while the new one takes fresh traffic, deploys and workflows stop competing for the same moment.

Don't make the work wait for the deploy — or the deploy wait for the work.

Blue-green is the classic shape of that idea: two environments, one live, one idle, flip between them. It got me most of the way. But two slots still mean a third deploy has to wait for a slot to drain. Generalising to one version per deploy removed that last constraint.

03 · The solutionA release lifecycle in six steps

  1. Boot a fresh version. Each deploy starts a new Compose project named <sha>-<timestamp> on its own port, running uvicorn with 4 workers.
  2. Gate on health. Nothing gets traffic until the new version passes its health check — up to 60 attempts, 2 seconds apart.
  3. Switch gracefully. The deploy rewrites nginx's upstream to the new port and reloads nginx gracefully, so existing connections finish on the old workers.
  4. Drain, don't kill. The previous version keeps running for a drain window — 2 hours, raised to 3 for production — so in-flight workflows complete.
  5. Reap on a schedule. A cron janitor runs every 15 minutes and removes versions whose drain window has passed.
  6. Roll back in one step. The previous upstream is recorded in a .prev file; rolling back is pointing nginx at it again.

04 · ImplementationThe deploy, and three bugs worth the price of admission

The deploy, in miniature

# deploy.sh — simplified
exec 9>deploy.lock && flock 9                  # one deploy at a time
release="$(git rev-parse --short HEAD)-$(date +%s)"
port="$(next_free_port)"

docker compose -p "$release" up -d --build     # new version, beside the old ones
wait_healthy "$port" 60 2                      # 60 tries × 2 s, or abort

cp upstream.conf .prev                          # remember where we were
render_upstream "$port" > upstream.conf        # point nginx at the new port
nginx -s reload                                 # graceful: old connections finish

Simplified — helper functions and error handling omitted.

Bug 1 · The health check that never passed

GotchaThe first version of the gate looked for pretty-printed "status": "ok". FastAPI returns compact JSON — {"status":"ok"} — so a perfectly healthy release never passed. Parse the JSON; don't grep it.

Bug 2 · The lock held hostage

GotchaDeploys serialise on a flock held on file descriptor 9. A git credential helper started during the deploy inherited that descriptor and outlived the script — keeping the lock alive and blocking every deploy after it. The fix was a few characters of shell: close fd 9 for the child with 9>&-.

Bug 3 · The janitor with the wrong clock

GotchaThe first janitor reaped versions by container age. But a version that's been live for hours is "old" the moment it's replaced — exactly when it still has in-flight work. It now measures time since replacement, which is what the drain window is actually about.

05 · ResultsDeploys became a non-event

  • Deploys no longer wait for running workflows. In production we've watched two versions serve at the same time — the new one taking every new request while the old one finished its work.
  • No request is routed to a release that hasn't passed its health gate.
  • Rolling back is a pointer swap, not a rebuild.
  • The drain window is one setting, tuned to how long production workflows actually run — 3 hours today.

06 · LessonsWhat I took away

  • Your deploy strategy is part of your application architecture. If the app holds in-process work, the deploy has to respect it.
  • Two slots are a special case. When deploys can outpace drains, generalise to N.
  • The bugs live in the edges — string matching, inherited file descriptors, the wrong clock. Test the unhappy paths on purpose.

Fighting deploys that step on running work? I'd love to hear how you're handling it.