LLM observability changes when a model call becomes part of a production workflow. A request may retrieve private material, invoke tools, pass through policy checks, generate a draft, and update a business record. A request log can show that a call happened; it cannot explain the user-visible outcome or the path that led to it. Production LLM observability should let an authorized investigator answer what the system attempted, which evidence it used, which controls ran, what the user received, and whether the intended work was completed. OpenTelemetry provides shared telemetry concepts, while the product team must define meaning, retention, access, and ownership. Compare the decision with embeddings before the first build and semantic search in production.
Define the outcome contract
Start with a workflow outcome rather than token counts. A support-drafting assistant may be expected to produce a source-grounded draft that reduces reviewer effort without inventing a policy. A tool-using agent may be expected to classify a request, propose an action, and stop when authorization is missing. Record the user-visible result, business state, required evidence, relevant version identifiers, failure categories, and owner. The OpenAI evaluations guide is useful for turning desired behavior into repeatable tests. An outcome contract keeps telemetry connected to a decision a team can improve.
| Trace layer | What to record | Why it matters |
|---|---|---|
| Request | Tenant scope, request class, policy version, and correlation ID. | Defines who and what the workflow served. |
| Evidence | Retrieval query, selected references, and source versions. | Shows the context available to the model. |
| Action | Tool proposal, authorization, execution result, and actor. | Separates intent from a real side effect. |
| Outcome | Validation, response, user correction, and final state. | Connects model behavior with work completed. |
Trace the complete workflow
Instrument boundaries instead of treating the model invocation as the whole product. Propagate a correlation identifier across the client, gateway, retrieval service, model runtime, tool service, policy engine, and system of record. Use consistent names for service, operation, model route, prompt or instruction version, retrieval collection, and outcome class. OpenTelemetry semantic conventions improve interoperability, but do not force every payload into a shared schema. Keep business meaning in carefully designed attributes and links. A responder should be able to follow sequence and timing without receiving an uncontrolled copy of every conversation.
Trace design should distinguish proposed work from executed work. For example, a model may propose cancelling an order, the policy engine may reject it, and no cancellation should be recorded as having occurred. Persist that distinction. If a human edits a generated draft, capture the review result and the reason category where appropriate. This is more valuable than logging a single final text because it reveals where the workflow failed: retrieval, reasoning, authorization, validation, or user interface. The NIST Generative AI Profile supports lifecycle evaluation and incident disclosure, both of which require this kind of evidence.
Measure outcomes, not just tokens
Operational signals should combine system health and work quality. Track latency by route, error and timeout rate, retrieval availability, tool rejection, policy rejection, escalation, reviewer correction, user retry, and task completion. Add quality sampling for groundedness, completeness, and harmful or unauthorized behavior. A low latency trace can still represent a failed task; a longer trace may be acceptable when it avoids rework. Treat evaluation cases as a maintained asset with representative, adversarial, and newly observed examples. Compare a proposed change with a baseline before promotion, and retain enough context to explain why the decision was made.
| Signal family | Example measure | Interpretation |
|---|---|---|
| Reliability | Timeouts, retries, and dependency failures by route. | Can the service complete the workflow consistently? |
| Grounding | Supported answer rate and source coverage. | Did the result use permitted evidence? |
| Safety | Blocked tools, policy violations, and unsafe outputs. | Did controls stop risky behavior? |
| Human value | Correction rate, review time, and completion. | Did the system improve the actual task? |
Protect telemetry as production data
Prompts, retrieved passages, tool arguments, and generated outputs can contain personal, confidential, or commercially sensitive information. Minimize payload capture, redact before export, separate content from operational metadata, and enforce role-based access. Keep short-lived high-detail traces for approved investigation while retaining aggregate measures longer when that serves the purpose. Make sampling explicit so an incident investigator knows what evidence may be missing. The OWASP LLM guidance is a useful reminder that prompt injection and insecure output handling can turn observability into another attack surface. Never let debug logging become an unreviewed data copy.
Use observability in release and incident work
A production trace earns its place when it changes a decision. Before releasing a model, prompt, parser, retrieval index, or policy, compare representative evaluations and key operational signals. During an incident, use the trace to separate model behavior from upstream data changes, tool permissions, latency, and user-interface defects. Keep a stable release identifier and show when a source collection or instruction changed. The LLM observability guide complements this production focus, while the model evaluation guide helps connect traces to quality review.
Production readiness also depends on a clear evidence budget. Decide which traces are retained for routine operations, which are sampled for quality review, and which require an approved temporary capture. Show sampling and redaction state in the trace so a responder knows whether an apparent gap is a system failure or an intentional privacy choice. When the service serves multiple tenants, make scope visible without exposing neighboring customers. Good trace context narrows the investigation while disciplined minimization narrows the access risk. This boundary belongs in the service review record.
Connect telemetry to ownership at the point of failure. A retrieval error should route to the source or index owner; an authorization rejection should route to the policy owner; a downstream write failure should identify the system-of-record team. Do not make the on-call engineer infer responsibility from a service name. Store runbook links and escalation conditions with the relevant operation. This shortens the distance between an observation and a safe action, which is the real value of production observability. This boundary belongs in the service review record.
Review traces for user impact, not only technical correctness. A response may satisfy a schema while missing the customer’s question, or a tool may execute successfully while creating duplicate work. Sample completed, corrected, refused, and failed cases. Ask a domain reviewer to classify the result and compare that classification with automated signals. Those disagreements are valuable evidence for better evaluation cases, routing rules, and product design. This boundary belongs in the service review record.
When retiring a model or workflow, preserve enough history to explain past decisions without keeping unnecessary content forever. Record the final version, known limitations, open incidents, and replacement route. Remove credentials and unused collectors, update runbooks, and verify that old dashboards no longer imply a live dependency. Retirement is part of production observability because stale traces and stale ownership can mislead responders during the next incident. This boundary belongs in the service review record.
Set a retention tier for each model-assisted workflow rather than applying one trace policy everywhere. A low-impact drafting route may keep compact metadata and sampled excerpts, while a workflow that changes access or money needs stronger evidence, controlled replay, and a named reviewer for temporary capture. Record the reason for the tier beside the route so a later privacy review can distinguish deliberate minimization from an instrumentation gap.
Map failure ownership by the first boundary that can change the outcome. A retrieval mismatch belongs with the corpus or index owner; an authorization rejection belongs with the policy owner; and a downstream write error belongs with the system-of-record team. Put those destinations in the trace contract and run a deliberately failed request to prove the handoff works under pressure.
Quality review should sample trajectories that look successful as well as those that error. Ask whether the model saw the permitted evidence, whether the tool call matched the approved intent, and whether the final response preserved the customer’s requested constraint. Compare reviewer judgments with automated checks, then turn disagreements into cases that exercise the observability design itself.
When a model route is retired, preserve the evidence needed to explain decisions made during its active window and remove the paths that could still emit misleading telemetry. Revoke collectors, update dashboards and runbooks, mark historical schemas clearly, and confirm that a responder can tell the difference between an archived trace and a live dependency.
Use a model-specific trace budget for high-volume inference. Keep route, version, policy result, tool outcome, and timing for every request, then sample sensitive content only when an approved investigation requires it. This preserves the ability to compare model behavior without turning routine observability into an uncontrolled transcript store.
When a tool call is denied, record the policy decision and safe reason code rather than only the model’s error text. That lets the team separate a correctly blocked action from a broken integration. For retrieval, capture collection and freshness metadata; for writes, capture the target record class and idempotency result without copying the protected payload.
Privacy review can compare observability fields with the workflow’s actual decision needs. Remove attributes that never change routing or diagnosis, hash identifiers only when correlation remains useful, and restrict cross-tenant joins. The goal is a trace that answers the operational question while making accidental reconstruction of a person’s conversation difficult.
Sunset a collector with an explicit comparison window. Announce the replacement, verify that alerts and incident exercises use the new fields, and retain a bounded bridge only while consumers migrate. Once the bridge expires, delete stale credentials and access rules so an old dashboard cannot quietly become an unsupported production dependency.
Key takeaways
- Define the user and business outcome a trace must explain.
- Propagate context across retrieval, model, policy, tool, and final-state boundaries.
- Record proposals separately from authorized and executed actions.
- Measure task completion, grounding, safety, and review effort alongside latency.
- Treat prompts, retrieved text, and outputs as protected production data.
- Use traces to compare releases and reconstruct incidents, then improve the workflow.
Frequently asked questions about LLM observability
Should every prompt be stored? No. Store only what the purpose, retention policy, and access model justify, and prefer references or redacted samples. Is model latency the main production metric? It matters, but completion and safety determine whether the workflow is useful. Do OpenTelemetry conventions solve LLM observability? They provide shared vocabulary and propagation patterns; teams still need outcome and privacy contracts. How much evaluation is enough? Enough to cover the important workflow, known failure modes, representative inputs, and changes observed in production, with a maintained review set.
Conclusion
LLM observability in production is a decision system, not a larger log stream. Trace the full workflow, protect its sensitive context, measure what users are trying to accomplish, and make release and incident evidence comparable. When the next investigation can distinguish a bad source, a rejected action, a slow dependency, and a model regression, observability has become operationally valuable.