Here is a failure mode that doesn’t show up in any dashboard. An AI feature ships, works well, and six weeks later the answers are noticeably worse. Nobody can point to when it slipped, because nobody was watching the one thing that changed. There was no outage, no error spike, no alert. The quality just drifted down a few percent a week until a customer finally said something.
LLM observability is the practice of logging what your AI feature actually returns in production and checking whether those answers are still good, not just whether the request succeeded. “It worked when we shipped it” is a statement about the past. Observability is how you make a statement about the present. It’s one discipline in a production-ready AI system, and it decides whether you find a problem in your metrics or in your inbox. It rests on the same foundations as everything else that keeps an AI feature working after launch.
Why does a shipped AI feature drift?
A feature can pass every test you wrote and still get worse in production, because three of the things that decide its output keep moving after launch. None of them throw an error.
The inputs change. Real users find phrasings, use cases, and edge cases your test set never imagined. The feature was tuned for the inputs you anticipated, and the world keeps sending new ones. The model isn’t failing so much as being asked questions it was never measured on.
The model changes underneath you. If you call a provider by an undated name, that name points at whatever version the provider ships this month, and providers do retire and replace models on a schedule. Pinning a specific version buys you stability, which is one more reason to keep the model behind a clean contract, but even a pinned version eventually gets deprecated and forces a move. This isn’t hypothetical: researchers who tracked a major model across two versions three months apart found its accuracy on one task had dropped from 84% to 51%, with no error and no announcement. Either way, the model you tested is not guaranteed to be the model running next quarter.
Behavior surfaces at scale. The demo ran a handful of inputs and the eval suite ran a few hundred. Production runs hundreds of thousands, and volume finds the rare failure that a smaller sample never will.
Why uptime monitoring misses AI failures
Most teams already monitor their systems, and reasonably assume that covers the AI feature too. It doesn’t, because uptime and performance monitoring watch the wrong layer.
An APM dashboard sees that the request returned in 800 milliseconds with a 200 status code. The endpoint responded, no error fired, and the monitoring dashboard shows green, which is the only sense in which the feature is “working.” The failure lives inside the response body, in whether the answer was actually any good, and infrastructure monitoring has no opinion on that. This is the line between DevOps monitoring and LLM observability: one confirms the model replied, the other tells you whether the reply was worth sending.
Picture a support assistant that starts quoting outdated refund policy after a provider updates the model underneath it. Every one of those calls returns a 200 in under a second, so the dashboard stays green the whole time the feature is confidently telling customers the wrong thing. Uptime monitoring is doing its job correctly and still tells you nothing, because the failure was never an error in the first place.
What to log for LLM observability
Observability starts with recording enough about each call to reconstruct a bad answer later. For every model call, log the input, the output, the latency, the token counts, the cost, the model and version that served it, and a trace id that ties it back to the user action. When someone reports a weird answer, that’s the difference between finding the specific call in minutes and never finding it at all.
The trace id is the piece that earns its keep. Say a user reports that the assistant gave them the wrong delivery window on Tuesday afternoon. With a trace id carried from the user action through to the model call, you find that exact request in the log, read the prompt the model actually received, see the context it was working from, and watch where the answer went wrong. Without it, you’re working backward from a screenshot and a rough timestamp, replaying inputs by hand and hoping to reproduce something that might happen one call in a thousand.
This does mean you’re storing user inputs and model outputs, which is a privacy decision, not just an engineering one. Scope what you keep, redact what you don’t need, and treat the logs as the sensitive data they are. The goal is enough visibility to debug the feature, not a permanent record of everything anyone ever typed.
Sampling and human review
You can’t read every call, and you don’t need to. Take a representative sample, pull in everything that tripped a guardrail or got a thumbs-down, and have a human review that slice against the same criteria your eval suite uses. Sampling is the affordable version of watching. It won’t catch every bad answer the moment it happens, but it will surface a downward trend long before the aggregate becomes a churn number.
The signals that tell you AI quality slipped
Three signals, in rough order of how directly they speak. Watch for movement in any of them.
Explicit feedback. A thumbs up or down, a “this isn’t what I asked for,” a correction. The most direct signal, and the rarest, because most users don’t bother. Worth collecting anyway.
Downstream behavior. The user re-asked the same question a different way, ignored the suggestion, abandoned the flow, or escalated to a human. Nobody clicked a feedback button, but the behavior says the answer missed. These implicit signals are quieter than a thumbs-down and far more common, so they often move first.
Periodic eval runs against production traffic. Point the eval suite at a sample of real inputs on a schedule, not just when you change something. If the same suite that passed at launch starts failing on this week’s traffic, the model or the inputs moved, and you now know roughly when. This is where observability and evals meet: the suite that gates your changes also becomes a standing monitor.
LLM observability vs. measuring AI search traffic
Because “measuring AI” is an overloaded phrase right now, worth being precise. This post is about the health of an AI feature you built and run. Measuring whether AI search engines can see and cite your website is a different problem with different signals, and we covered that one separately in how to measure AI search traffic. Same word, different job.
How much observability a feature actually needs
Observability costs money and attention. Sampling and human review take time, storage isn’t free, and the privacy surface is real. So right-size it to the stakes. A low-traffic internal tool needs a fraction of what a customer-facing assistant needs, especially one that can make commitments on the company’s behalf. What gets teams in trouble is spending nothing on a load-bearing feature and calling the green uptime dashboard “monitoring.”
What observability actually buys you
Observability is what turns “the model got worse at some point” into “quality dropped on the fourteenth when the provider shipped a new version, and here’s the eval diff that proves it.” One of those is a mystery you burn a week on. The other is a Tuesday afternoon.
It’s the layer that assumes a working AI feature will eventually stop working quietly, and makes sure you’re the one who notices first. We build it into the AI features we ship and the ongoing monitoring around them, because launching an AI feature is only half the job.
If you’ve got an AI feature in production and you’re not sure how you’d know if it started slipping, tell us what you’re building. We respond within one business day with the gaps we see and a rough scope for closing them.