Agent observability is a distinct problem from LLM monitoring — failures show up in multi-step causal chains across turns and tool calls, not at the level of a single API call.
Core concepts
- Why agent-native tracing matters: Tools built for single-prompt LLM logging capture independent events you must manually correlate; agent-native tools capture the causal chain between steps directly.
- Langfuse: The most widely deployed open-source option, self-hostable, strong for teams with data-residency requirements. Langfuse Docs
- LangSmith: Near-zero setup if you're already on LangChain/LangGraph; more friction outside that ecosystem. LangSmith Docs
- Braintrust: Eval-led, a16z-backed, the most generous free tier and the strongest CI/CD eval-gated deployment workflow of the major platforms. Braintrust Docs
- Arize Phoenix: OpenTelemetry-native, open-source, a strong pick if you want standards-based instrumentation rather than a vendor-specific SDK. Arize Phoenix Docs
- Evals: Automated scoring of agent output against a dataset — the difference between "it worked in the demo" and knowing it works across a representative sample of real inputs.
What Is It?
Tracing tools record hierarchical spans for each model and tool step.
Eval platforms score outputs against golden sets and rubrics.
Together they close the loop on agent quality in CI and prod sampling.
Why Should I Care?
Multi-agent bugs show up three hops deep — flat logs hide causality.
Evals prevent 'vibes-based' prompt edits in teams.
Real World Analogy
Traces are GPS breadcrumbs; evals are the driving test — breadcrumbs show where you went, the test says if you should get a license.
Visual Mental Model
Langfuse / LangSmith / Braintrust
spans: LLM -> tool -> LLMBeginner Explanation
Use Langfuse, LangSmith, Braintrust, or Phoenix to capture spans and run eval datasets when prompts or tools change.
Smallest Possible Example
def score(answer: str) -> float:
return 1.0 if "expected" in answer else 0.0Common Mistakes
❌ Eval sets that never update.
Product changes; stale evals give false confidence.
❌ Tracing only in prod.
Dev reproduction needs the same spans locally.
❌ One metric (BLEU) for conversational agents.
Use task success, tool accuracy, and human rubrics.
Real Project Example
Arize Phoenix helps debug retrieval and embedding issues visually.
Teams on OpenAI APIs export traces to third-party eval vendors.
Build This Mini Project
Create five eval cases for a FAQ bot (question, expected substring); run a script scoring current answers.
Interview Question
What belongs in an agent trace span?
Step name, timestamps, model/tool identifiers, inputs/outputs (redacted), errors, token usage, and parent/child links to reconstruct the full loop.
Key Takeaways
- Spans show causal chains.
- Eval datasets gate prompt changes.
- Refresh evals with product.
- Trace in dev and prod.
Resources
YouTube learning
Core concepts from video walkthroughs:
The 60-second story
A single-prompt log is a photo of one phone call. An agent is a shift: a call, a tool, a bad argument, another call, a handoff. If your photos are not clipped together, you will stare at a tool error and not know which prompt caused it. Agent-native tracing is the string that ties the photos into one shift.
Langfuse is the open-source clipboard a lot of teams can self-host when the photos are not allowed to leave the building. LangSmith is the clipboard that is already in the drawer if you live in LangChain or LangGraph, and more of a project if you do not. Braintrust cares first about exams: score this shift against a set of cases, and do not deploy when the score drops. Phoenix speaks OpenTelemetry, which matters if your company already standardized on one kind of wire for traces and does not want a new religion.
Evals are the practice exams. “It worked when I tried it” is a single photo. An eval set is fifty guests with known allergies. You notice the agent recommended the peanut sauce before the guest does. That is the difference between a demo and a release.
Map the shift.
- One LLM log line is a photo.
- A trace is the string through the whole shift.
- Langfuse is a clipboard you can keep in-house.
- LangSmith is the clipboard next to LangGraph.
- Braintrust is the exam proctor.
- Phoenix is the standard wire.
- An eval is a room of known test guests.
trace_id ties: user message -> model -> tool -> model -> final
score(trace, expected)Without the id, you have anecdotes. With the score, you have a gate.
These tools exist because agent failures are causal. The last token is not the bug. The tool result two steps back is the bug. Evals exist because production is too late to discover you regressed the refund policy. Pick a clipboard your team will actually open. An unread trace vendor is an expensive diary.
Mental model: photos on a string, then a practice exam. Debug the string. Ship when the exam still passes.
You have the page if you can say why a prompt log is not an agent trace, and what an eval protects you from that a happy demo does not. When the shift goes wrong, you want the string of photos, not one flattering picture of the last sentence.
