Manas Mishra
Résumé

Case study Datavio · Data engineering

Queue it, fan it out: a pipeline that stopped blocking the app

Heavy processing used to run inside the app, one job after another. Moving it onto SQS and Lambda let the work run in parallel — and let the API get back to answering requests.

Role
Designed & built
Context
Datavio · data platform
Stack
AWS SQS · Lambda · S3 · Django
Pattern
Queue-based fan-out
Faster processing once microservices ran in parallel instead of in sequence
2×
The same four jobs: run in sequence inside the app, versus fanned out across Lambda workers from an SQS queue — finishing in half the time.
TL;DR Data-processing work used to run inside the application's own services, one step after another. Now Django APIs enqueue units of work on AWS SQS and Lambda functions pick them up in parallel, with S3 holding the data in between. Running the microservices in parallel made processing 2× faster, and heavy jobs no longer hold up user-facing requests.

01 · The problemOne slow step held up everything behind it

Processing jobs ran synchronously inside the app's services, in sequence. Big batches took a long time, occupied workers that should have been answering users, and a single slow step delayed everything queued behind it.

02 · The intuitionMost of the work didn't depend on itself

Look at the steps and most of them were independent — they only ran in sequence because that's how they'd been written. Put each unit of work on a queue and let as many workers as needed pull from it: the queue absorbs spikes, failed work gets retried, and the app just enqueues and moves on.

The app's job is to answer requests. Everything else goes on a queue.

03 · The solutionEnqueue, fan out, retry

  1. Enqueue, don't execute. Django APIs publish work items to SQS and return immediately.
  2. Fan out on Lambda. Lambda functions consume the queue and scale out with the backlog, processing items in parallel.
  3. Keep data in S3. The heavy data lives in S3; the queue carries the work.
  4. Retry by default. If a worker fails, its message becomes visible on the queue again and is retried — no bespoke retry code in the app.

04 · ImplementationChoices that shaped it

Picking the queue setup

I experimented with a few queue-based setups — including SQS with Lambda, and FastAPI-based workers — before settling on the shape that fit: SQS for durable, decoupled queueing and Lambda for elastic execution without servers to manage.

The unit of work is the design

Queue pipelines live or die by their work items. Small, independent items that any worker can process in any order are what let the pipeline scale out instead of up.

Before

Jobs run in sequence inside the app. Long batches tie up app workers, and one slow step delays the rest.

After

Jobs fan out across Lambda. The app enqueues and returns; slow items no longer block fast ones.

05 · ResultsFaster, and out of the way

  • 2× faster processing by running microservices in parallel instead of in sequence.
  • Heavy processing moved off the request path, so it no longer blocks application performance.
  • Large-scale data processing with minimal latency, on a system that's more scalable and resilient.

06 · TakeawaysRules of thumb I still use

  • If it doesn't need to happen during the request, it shouldn't.
  • Queues turn "faster" into "wider" — parallelism is often cheaper than optimisation.
  • Retries belong in the infrastructure, not in every function.

Untangling a pipeline that's blocking your app? Always happy to talk queues.