Who is writing this, and why?
The last sentence of the definition is the whole argument of this page, so it is worth being clear about where it comes from. Overmind is a model training platform for AI teams: it turns production traces into specialised models a team owns, trained, benchmarked and served. Observability is the first layer of that loop rather than the endpoint. Overmind ingests OpenTelemetry from anywhere and has a direct interest in you having good traces. What follows is about making traces useful, whichever tool you collect them in.
What does LLM observability capture?
A trace is the set of spans sharing one trace ID. The root span is the end-to-end run: one user request, one agent invocation, one scheduled job. Its children are the individual steps.
For an LLM application, each span carries more than a normal HTTP span does:
| What the span records | Why it exists | Typical attribute |
|---|---|---|
| The rendered prompt | The template is not the prompt. Variables, retrieved context and history are what the model actually saw | gen_ai.input.messages |
| The completion | The only place the model's actual words survive | gen_ai.output.messages |
| Model and version | Providers change weights behind a stable model name | gen_ai.request.model, gen_ai.response.model |
| Token counts, in and out | The unit your bill is denominated in | gen_ai.usage.input_tokens, gen_ai.usage.output_tokens |
| Cached tokens | Prompt caching changes cost by an order of magnitude | gen_ai.usage.cache_read_input_tokens |
| Cost | Either stamped by the client or derived from tokens and model pricing | genai.cost |
| Tool name and arguments | Which tool the model chose, and what it passed | gen_ai.operation.name = execute_tool, tool.name |
| Latency and status | Where the time went and whether the step errored | OTel span timing and status |
| Session identity | Which multi-turn conversation this run belongs to | conversation.id |
| Agent identity | Which of your agents this is, across deploys | resource attribute, e.g. overmind.agent.name |
Put those together and you get the thing every vendor demos: a waterfall of the run, expandable to the exact text of every model call. That view is useful and it is solved. Most tools in the category do it well.
How is LLM observability different from traditional observability?
Classic observability was built for deterministic systems. The same input produces the same output, so a failure is an exception, a bad status code, or a latency spike. LLM applications break every one of those assumptions.
| Traditional observability | LLM observability | |
|---|---|---|
| Failure looks like | An error, a timeout, a 500 | A confident, well-formatted, wrong answer that returns 200 |
| "Correct" is | Defined by a schema or a test | A judgement call, often without a reference answer |
| The unit of work | A request or a transaction | A trajectory: many model calls, tool calls and retries towards one goal |
| Cost driver | CPU, memory, egress | Tokens, and they scale with usage rather than with headcount |
| Repeatability | Same input, same output | Same input, different output, sometimes materially so |
| The dangerous state | Down | Up, fast, cheap, and quietly wrong |
| Instrumentation | Mature, mostly automatic | OTel GenAI conventions are still stabilising, and dialects disagree |
The three pillars of classic observability, metrics, logs and traces, still apply. They are just not sufficient on their own, because none of the three has an opinion about output quality. A dashboard that is green across metrics, clean across logs and complete across traces is entirely compatible with an agent that has been doing the wrong thing for a week.
What is the difference between LLM monitoring, observability, evaluation and optimisation?
These four words get used interchangeably by vendors, including by us in careless moments, and the distinction is the most useful thing on this page.
| Question it answers | Runs | Output | Needs a ground truth? | |
|---|---|---|---|---|
| Monitoring | Is it up, fast and inside budget? | Continuously, on aggregates | Alerts and dashboards | No |
| Observability | Why did this specific run do that? | On demand, per trace | Evidence: spans, prompts, tool calls | No |
| Evaluation | Was the output any good? | On a dataset, or on live traces | A score, a pass or a fail, and a reason | Yes, eventually |
| Optimisation | How do I make it better? | On a change, against a baseline | A prompt diff, a code diff, or new model weights | Yes |
Monitoring tells you the agent is alive. Observability tells you what it did. Evaluation tells you whether that was acceptable. Optimisation is the only one of the four that changes anything.
Most teams buy the first two, build a bit of the third by hand, and never reach the fourth. That is not a tooling failure so much as a sequencing habit: the first two are the ones that page you at 3am, so they get bought first.
What is agent observability, and how is it different?
"Agent observability" is the term the conversation is moving to, and it is not just a rebrand. The difference is what counts as the unit of work.
| LLM observability | Agent observability | |
|---|---|---|
| Unit | One model call | One trajectory, from goal to outcome |
| Question | What did the model say? | Why did the agent take this path? |
| Structure | A list of calls | A tree of decisions, tool calls, retries and delegations |
| Typical failure | A hallucinated fact | A correct tool called on a wrong premise, three steps back |
| Extra state | None | Sub-agent handoffs, memory reads and writes, sandbox lifetime, partial commits |
Tracing a single agent run is largely solved. What is not solved is the run with forty-seven steps, partial state committed halfway through, and a sub-agent that returned a plausible summary of work it never did. Practitioners describe this consistently: tracing is fine until roughly twenty agents call each other, at which point, as one engineer put it in an r/AI_Agents thread on enterprise agent platforms in August 2026, "the observability breaks down, you get lost in the noise."
If you are instrumenting today, instrument for the trajectory. Give spans a type (entry point, workflow, tool call, function, model call, retrieval), stamp a stable agent identity on every span, and group multi-turn exchanges under one conversation ID. Auto-instrumentation alone gives you a flat list of model calls, and a flat list cannot show you a decision path.
Why do agents fail silently even with full observability?
Because the failure modes that hurt do not produce a signal. Three of them recur constantly in production teams' own accounts.
The silent success. The agent returns a perfectly formatted completion summary describing work it did not do. Every span is green. Every status is 200. One engineer on r/AI_Agents described running a pipeline for a week before noticing the agent had been returning "task complete" with nothing underneath it. Nothing in the trace is wrong; the trace faithfully records a lie.
The run that just stops. A machine dies mid-run, a credential expires, a network call is blocked. The trace simply ends. There is no error span, because nothing errored, and there is no completion, because nothing completed. It looks identical to a run still in flight. One production team put it precisely: ask any vendor what they show you when a run dies silently instead of erroring.
The correct step on a wrong premise. The agent calls the right tool with well-formed arguments, and the arguments encode a misunderstanding formed four steps earlier. Schema validation passes. Tool selection metrics pass. The output is wrong for reasons no per-span check can see, because the error is in the relationship between spans.
The pattern behind all three: a trace is evidence, not a verdict. Observability records. It cannot grade, because grading requires a definition of good, and that definition lives in your product, not in your telemetry.
This is also why "just add an alert" does not close the gap. You can alert on error rate, latency and spend, and all three will stay flat through every failure above.
What should you do with the traces once you have them?
The honest answer is that the traces are an asset and most teams treat them as an archive. Four things turn them into something that changes behaviour, in rough order of effort.
1. Score them as they arrive, not when you go looking. Attach evaluators to root spans so every production run gets graded on arrival, at whatever sampling rate you can afford. This turns the trace list into a quality monitor: filter to low scores in the last hour and read exactly what failed. In Overmind this is live scoring, and evaluators come in six kinds, from deterministic rules (schema validity, tool selection, latency, cost) through model judges to trajectory and dataset-level scorers. Deterministic checks are free and instant, so send every mechanical question there and reserve the judge for the genuinely subjective ones.
Two details make live scores trustworthy rather than decorative. Outcomes are typed: every result is scored, abstained, not_applicable, skipped or error, and only scored feeds averages, so an abstention never becomes a fake number. And runs that cannot be faithfully reconstructed are marked degraded and excluded rather than quietly polluting the aggregate.
2. Turn failures into a dataset instead of a Slack thread. Select the traces that went wrong and promote them into a dataset, with each row keeping a link back to its source trace. That provenance matters more than it sounds: six weeks later, when a metric moves, the difference between "we think this row came from a real incident" and "here is the exact run" is the difference between a decision and an argument. How to turn traces into a training dataset walks through the selection step.
3. Pin the dataset as a regression set. Once failures are a dataset at a recorded commit, a prompt change can be run against them before it ships rather than after a customer finds it. The failure you already had is the cheapest test case you will ever get. This is the job of agent testing.
4. Then change something. Either the prompt and tool logic, which an optimiser can rewrite and open as a reviewable diff, or the model itself, fine-tuned on the trajectories your agent actually produces and benchmarked against your current production model using your own evaluators. On our own benchmark a 0.5B specialist trained this way scored 50% against GPT-4o mini's 29% on the same product evaluation.
None of steps 1 to 3 requires you to leave your current observability tool. Steps 1 and 2 are worth doing even if you never get to step 4.
Do you need OpenTelemetry for LLM observability?
You do not strictly need it, and you should use it anyway.
OpenTelemetry is the only thing in this category that is vendor-neutral. Instrument with OTel and your traces are portable; instrument with a proprietary SDK and your history is hostage to a pricing page. The GenAI semantic conventions are still stabilising, which is a real cost, but the alternative is worse.
The practical wrinkle is dialects. There are several competing attribute vocabularies in circulation: the OTel GenAI semantic conventions, OpenLLMetry/Traceloop keys, OpenInference, and each vendor's own. They disagree about something as basic as what to call an input token count. Any backend worth using should read more than one. Overmind's ingest reads three dialects in priority order, its own SDK keys, OTel GenAI, and OpenLLMetry/Traceloop, and normalises Traceloop scope names on the way in. OpenInference attributes are not parsed today, which is the kind of thing worth checking before you commit rather than after.
Getting spans in should not require infrastructure. Overmind takes OTLP/HTTP protobuf posted straight to the API with an API key header, with no collector, sidecar or proxy in between, so a stock OTel exporter works with nothing but a base URL:
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.overmindlab.ai"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=ovr_..."Or, in Python, with auto-instrumentation for the OpenAI, Anthropic, Gemini and Agno clients:
import overmind
overmind.init(
service_name="research-agent",
environment="production",
agent_name="Research Agent",
)Trace ingest is deliberately exempt from credit gating, because telemetry you stop sending when you run out of budget is telemetry you cannot trust.
How do you choose an LLM observability tool?
The ranked lists of tools are easy to find and not very useful, because every product in the category passes the feature-checklist test. These are the questions that actually separate them, drawn from what production teams say they wish they had asked.
- What do you show me when a run dies silently instead of erroring? The single best question in the category. Most answers are unconvincing.
- Is it OpenTelemetry, genuinely? Can I point a stock OTel exporter at it and get sensible spans, or does it need your SDK? Which attribute dialects do you parse?
- Can I see cost next to the behaviour that caused it? Per-run spend tied to the specific tool call that looped, not a monthly total.
- Can a production failure become a regression case without copy-paste? If turning a bad trace into a test requires a human to retype it, it will not happen twice.
- Does anything score traces automatically, or do I only find failures by browsing? Browsing does not scale past the first thousand runs.
- Where does trace data live, what is the subprocessor list, and what happens to a trace containing user data when an erasure request arrives? Enterprise buyers ask this first and it is usually answered with a link to a docs page rather than an answer.
- What happens when my provider changes model weights behind a stable model name? If the answer is "you will see it in the dashboard", ask which number moves.
Ask us the same seven. On the sixth, the honest answer is that credentials and connector secrets are encrypted at rest and the rest is a conversation, not a certification: we do not claim compliance we do not hold. Start with the observability product page or the docs, then open a free project and send it a trace.
Common questions
What is the difference between LLM observability and monitoring?
Monitoring answers whether the system is up, fast and inside budget, using aggregates and alerts. Observability answers why one specific run behaved the way it did, using the trace of that run. Monitoring tells you something is wrong; observability helps you find out what. Neither tells you whether the output was correct, which is the job of evaluation.
Is LLM observability just APM for AI?
No, though it borrows the machinery. APM assumes deterministic systems where failure means an error or a latency spike. LLM applications fail most often while returning a 200 with a well-formed, confident, wrong answer. The extra work is capturing prompts, completions, token cost and multi-step trajectories, and then grading output that has no reference answer.
What are the three pillars of LLM observability?
The classic three, metrics, logs and traces, still apply, with traces doing most of the work because the unit of interest is a multi-step run rather than a single request. In practice teams add two more: token and cost accounting per run, and output quality scores. Without the last one you have complete records of behaviour you cannot judge.
What is the difference between LLM observability and agent observability?
LLM observability treats the model call as the unit of work. Agent observability treats the trajectory as the unit: the whole path from goal to outcome, including tool calls, retries, sub-agent handoffs and memory reads. Tracing an individual call is largely solved. Reconstructing why an agent chose a path forty steps ago is not.
Do I need OpenTelemetry for LLM observability?
Not strictly, but it is the most widely adopted vendor-neutral standard and it keeps your trace history portable. The caveat is that several attribute dialects are in circulation and they disagree on basic keys like token counts, so check which ones your backend parses before you commit. A backend that accepts OTLP directly, with no collector to run, removes most of the setup cost.
How much trace data should I keep, and does sampling break it?
Keep whole traces rather than sampled spans: half a trajectory tells you almost nothing about a run that went wrong. If cost forces sampling, sample at the trace level and keep 100% of runs that error, cost more than a threshold, or score badly. Uniform random sampling is the most likely to drop the rare, expensive failures you needed to see, so bias the sample towards them.
Can LLM observability tell me whether my agent is any good?
No, and this is the most common misunderstanding about the category. A trace records what happened with perfect fidelity, including when what happened was a confident lie. Judging quality requires a separate layer: deterministic checks for anything mechanical, model judges calibrated against human labels for anything subjective, and a definition of good that lives in your product rather than your telemetry.
What do I do with the traces once I have them?
Score them as they arrive so failures surface without browsing, promote the bad ones into a dataset that keeps provenance back to the source trace, pin that dataset as a regression set so the next prompt change is tested before it ships, and then change something: the prompt and tool logic, or the model itself, trained on the trajectories your agent actually produces and benchmarked against what you run today.