July 2026 · 12 min read
Last year, a friend's startup deployed an LLM-powered customer support bot. It worked fine in staging. Three weeks after launch, customers started complaining. The responses were still “correct” by any single-sample test you’d run: grammatically fine, roughly on topic. But something had quietly drifted. The model was giving shorter, vaguer answers. Nobody caught it because nobody was watching the sequence.
This is the problem I got obsessed with while building Continuum, my LLMOps platform. And it led me down a surprisingly old rabbit hole: Statistical Process Control, a technique invented in a Bell Labs factory in 1924.
Imagine you’re manufacturing steel rods. Each rod should be 10 mm in diameter. You have measurement noise, so you accept anything between 9.95 mm and 10.05 mm. If one rod measures 10.8 mm, you stop the line. That’s obvious.
But what if the rods are slowly drifting? Day 1: average 10.01 mm. Day 3: 10.03 mm. Day 7: 10.06 mm. Each individual measurement passes your quality check. But the trend is unmistakably heading toward reject territory.
Walter Shewhart at Bell Labs in 1924 formalized a solution: control charts. You track a process over time and apply statistical rules to detect non-random patterns before they become failures.
Now replace “rod diameter” with “LLM response quality score.” Same problem. Same solution.
A control chart has three horizontal lines:
For LLMs, we only care about the lower side. If the model suddenly gets better, that’s fine. We care when it gets worse.
Every time a new response comes in, you score it, add the score to the chart, and ask: does this point violate any of the control rules?