Case study Side project · AI infrastructure
llm-graph-optimizer: finding the LangGraph nodes that don't need an expensive model
A LangGraph agent's biggest cost is often a decision that should cost almost nothing. I built a tool that finds those decisions automatically — using your own production traffic as the evidence.
- LLM cost on one classify node ($4.20 → $0.38 a month)
- −91%
- Faster p50 latency (310 ms → 96 ms)
- 3.2×
- Agreement with the LLM at confidence above 0.8
- 98.7%
- Line to instrument a compiled graph — no other changes to the agent
- 1
$ lgo report
| node | agree | p50 ms | $/mo |
|---|---|---|---|
| classifyswap | 98.7% | 310→96 | 4.20→0.38 |
plan, respond · open-ended → keep on llm
✓ 1 node worth swapping · −91% cost · 3.2× faster
lgo report on the pilot graph: the classify node is a clear swap candidate, while the open-ended nodes stay on the LLM.lgo shadow replays those real inputs through a lighter model (Jev) in the background, and lgo report compares cost, latency, agreement, confidence and projected savings per node. In a small pilot, a classify node went from $4.20 to $0.38 a month and from 310 ms to 96 ms p50, agreeing with the LLM 98.7% of the time at confidence above 0.8.
01 · The problemEvery node gets the expensive model
Agent graphs grow one node at a time, and almost every node starts life the same way: call the big, general-purpose model. It's the safe default while you're building. The trouble is that the default rarely gets revisited once the agent works.
I kept seeing the same pattern in LangGraph agents. Nodes that did nothing more than classify, route or verify — pick a label, choose a branch, check an output — paid the same price and the same latency as the nodes doing real reasoning.
Your agent's biggest cost is often a decision that should cost almost nothing.
Those aren't open-ended generation problems. They're structured decisions with a small answer space, and they run on every single request. The cost per call is tiny; it compounds quietly with traffic.
Knowing this in theory is easy. Knowing which nodes in your graph are safe to move to a cheaper model — on your traffic, without degrading the agent — is the hard part. Guessing gets you either timid savings or silent regressions.
02 · The intuitionLet production traffic make the case
You don't have to guess, because the evidence already exists. Every time a node runs, it receives an input and the LLM makes a decision. Record those pairs and you can ask a precise question later: if a lighter model had seen exactly these inputs, how often would it have made the same decision — and how sure was it?
Agreement, filtered by confidence, turns "I think this node is simple" into a measurement. A node where the lighter model agrees almost every time is a swap candidate. A node where it doesn't stays on the LLM — and now you know why.
03 · How it worksInstrument, shadow, report
- Instrument. One line wraps the compiled graph. From then on, every node execution records its input, output and latency to a local SQLite file. The agent's own logic doesn't change.
- Shadow.
lgo shadowreplays the recorded inputs through Jev, a lighter model, in the background — never on the request path, so production latency is untouched. - Report.
lgo reportlines both models up per node — cost, latency, agreement, confidence and projected savings — so the swap candidates are obvious.
$ lgo shadow # replay recorded node inputs through Jev, in the background
$ lgo report # per node: cost · latency · agreement · confidence · savings04 · ImplementationSmall decisions that kept it useful
Recording stays local and boring
SQLite was deliberate: nothing to stand up, trivially inspectable, and easy to ship alongside any agent. The recorder captures exactly what's needed to replay a node in isolation later — its input, its output and how long it took.
Shadow mode, not an A/B test
Replays run out of band. Nothing the lighter model says reaches a user during evaluation, so you can measure aggressively without risk. It's the same principle as a shadow deployment: prove it on real traffic before you trust it with real traffic.
Per node, not per graph
Graph-level averages hide the story. One node can be trivially swappable while its neighbour absolutely isn't, so the report works node by node and every decision stands on its own evidence.
A safety net for the long tail
The swap doesn't have to be all-or-nothing. With a small fallback to the LLM endpoint for low-confidence cases, the swapped node kept the expected output in the pilot while the lighter model handled the easy majority.
05 · ResultsWhat the pilot showed
Before · classify on the LLM
$4.20 a month310 ms p50
After · classify on Jev + fallback
$0.38 a month96 ms p50 · 98.7% agreement at confidence > 0.8
- Cost: $4.20 → $0.38 a month for one classify node — about 91% lower.
- Latency: 310 ms → 96 ms at p50 — roughly 3.2× faster.
- Quality: 98.7% agreement with the LLM at confidence above 0.8; with the low-confidence fallback, the expected output didn't change.
The dollar amount on one node is small, and that isn't the point. The point is that the report uses your own production traffic to show which nodes are worth swapping — with numbers instead of intuition. Apply that to every classify, route and verify node, on every request, and the savings stop being small.
06 · TakeawaysWhat I'd tell anyone running agents
- "Default to the biggest model" is a build-time convenience, not a runtime strategy.
- Structured decisions — classify, route, verify — are the first place to look.
- Measure on real traffic, in shadow, before swapping anything.
- Keep a fallback: the cheap path handles the easy majority, the LLM handles the rest.