Error Analysis for LLM Applications - Step by step guide

This is a hands-on walkthrough through the five-step error-analysis process, applied end-to-end to a real example app: a phone tech-support chatbot. By the end you'll have a representative trace sample, an annotation queue with two passes of scoring, a clustered taxonomy of failure modes specific to the app, a chart of failure rates per category, and a decision for each — fix, measure with an evaluator, or just monitor.

If you want to learn more about error analysis first, you can check out our Academy page on Error Analysis

Two terms from qualitative research show up throughout: open coding (read traces, write free-text observations, no predefined categories) and axial coding (group those observations into named failure categories with a shared root cause).

Want to do this interactively?

Run with your agent

Paste this prompt into your coding agent. The Langfuse skill runs every step alongside you — pulling traces, clustering, computing failure rates. You make the domain calls.

I want to do a systematic error analysis of my LLM application to understand how it fails.
Please install the Langfuse skill (https://github.com/langfuse/skills/tree/main/skills/langfuse) and the Langfuse CLI (https://github.com/langfuse/langfuse-cli), then guide me step by step through error analysis.

The example app: Dad Tech Support

Throughout this guide we use a real example: a phone tech-support chatbot built for a parent who isn't comfortable with technology.

The app was built to help parents get phone help without calling every time something went wrong. The bot speaks as if it were the child: patient, non-technical. It can search the web for current info about the phone and carrier.

At the time of analysis: 505 traces across 478 sessions in Langfuse.

The process

The process has five steps, each building on the previous one. Ideally there are already ~100 traces logged in Langfuse.

Step 1: Gather a diverse dataset

In the first step you assemble a set of traces that represents how your app actually behaves, including both successes and failures. A diverse dataset surfaces all meaningful failure modes, not just the common ones.

Step 1.1: Choose what to annotate

Your traces are made of multiple observations. To target the correct one, you need to decide which unit to annotate before setting up the queue.

If your app is conversational, it often makes sense to annotate the last turn per session. It has the full conversation history in context. If your app is not following a chat interaction, you might want to select a specific observation that generates the final response of the agent.

Check the observation level. In OpenTelemetry-instrumented apps, trace-level input/output can be null. The actual content often lives in a GENERATION observation. Expand the observation tree in any trace to confirm.

In the Dad Tech Support example:

Trace: dad-chat-request
├── Root span
├── WebSearch (tool call, optional)
└── dad-chat-request [GENERATION]  ← annotate this
    input: full conversation history + system prompt
    output: bot's reply

Trace-level input and output were null. All readable content lived in the GENERATION observation. When adding items to an annotation queue, it often helps to target the GENERATION observation, not the trace.

Step 1.2: Select a representative sample

Target ~100 traces. The goal is coverage, not randomness: over-represent edges and anything already flagged as problematic. In the process of selecting traces there are certain signals worth looking for that qualify a trace to be part of the sample.

Signals to look for:

• Tags: If your app tags traces with error or flagged, include them.
• Existing scores: If you have a judge or human feedback scores, prioritize low-scoring traces.
• Latency: High-latency traces often involve tool use or complex reasoning.
• Cost: Very low-cost traces tend to be short refusals. Very high-cost ones tend to be verbose. Both worth including.
• Multi-turn sessions: Sessions with many turns are closer to real usage.

Browse and filter traces in Langfuse under Traces using the latency, cost, and tag filters.

No production data yet? Run your app against representative synthetic inputs, capture the traces in Langfuse, and sample from those. See the Langfuse datasets overview.

The Dad Tech Support sample (100 traces across 6 tiers):

One finding: of 478 sessions, only 11 were multi-turn. The rest were single-turn, possibly synthetic. Worth confirming scope before committing to a sample.

Step 1.3: Create an annotation queue

To get ready for annotating your data you create an annotation queue in Langfuse. For this, we first need to create score configs, then the queue and then add the respective examples.

Create two score configs (Settings → Scores → Create):

These two scores are fixed for every open-coding pass.

Always write a clear description for every score config. It appears next to the score field in the annotation UI. Without it, annotators guess.

Create the annotation queue (Annotations → Queues → Create):

Name it with the date and use case, e.g. 2026-04-16 Open Coding - Dad Tech Support. Add both score configs.

Add your sample to the queue:

In the Traces view, navigate to each trace's GENERATION observation and add it to the queue. You can also multi-select observations and add them in bulk. Target the observation, not the trace.

Step 2: Open code your first 30-50 traces

In the second step you review each trace in your example in detail. Work through the annotation queue in Langfuse. For each trace:

1. Read the full conversation in the observation view
2. Write what you observe in open_coding, plain language, no diagnosis
3. Set pass_fail_assessment to Pass or Fail

Rules for good notes:

• Describe behaviour, don't diagnose. Write "bot said it couldn't look up printer manuals despite the system prompt allowing web search", not "web search tool is probably broken."
• Focus on the first thing that went wrong. Errors cascade. Fix the root cause, not the downstream symptom.
• Don't start with a list of expected failures. A pre-defined list causes confirmation bias.

What the notes look like:

The first long session (12 turns) surfaced a revealing failure immediately. The system prompt said: "You are allowed to use WebSearch. Never say that you cannot look things up online." But the bot said "I can't look up printer manuals for you" twice, then capitulated after the user pushed back a third time. Direct contradiction between prompt and behaviour. This kind of finding only comes from reading real traces.

Stop reviewing when new traces stop revealing new kinds of failures. Rule of thumb: no new category in the last 20 traces. Around 100 total works for most apps.

Step 3: Cluster into failure categories

Once you have 30-50 notes, group them into categories. Goal: 5-10 distinct, named failure categories, each with a one-sentence definition clear enough that someone else could apply it consistently.

How to cluster:

1. Read through all failure notes
2. Group similar ones
3. Split notes that look alike but have different root causes
4. Merge notes that share the same underlying problem
5. Name each group and write a one-sentence definition

Rules for good categories:

• Split when root causes differ. "Bot hallucinated a settings icon" and "bot refused to search the web" both look like information problems, but one is a missing device lookup and the other is a prompt contradiction. Different fixes.
• Group when root causes are the same. Multiple notes about missing filters for different fields become one category: missing_query_constraints.
• Name after what broke. missing_device_lookup beats information_quality. identity_not_disclosed beats transparency.

LLM-assisted clustering:

LLMs can be a great help in coming up with the first version of your clustering. Paste your notes into your favorite coding agents with this prompt:

Here are failure annotations from reviewing an LLM pipeline.
Group similar failures into 5-10 distinct categories. For each:
- A clear name (snake_case)
- A one-sentence definition
- Which annotations belong to it

Annotations:
[paste your notes]

Always review the proposed groupings yourself. LLMs cluster by surface similarity and can produce groups that look plausible but conflate different root causes.

The Dad Tech Support failure taxonomy:

The LLM's initial clustering merged passive identity failure (didn't disclose being a support agent) with active impersonation (acted as the real child). Both are identity problems, but they have different root causes. The passive case needs a disclosure instruction. The active case needs the persona instruction dialed back. User review caught this.

After two rounds of refinement:

Step 4.1: Label all traces against the categories

Create one boolean score config per failure category (Settings → Scores → Create, type: Boolean).

Write a clear description for each, one sentence explaining what true means:

"True if the bot gave generic guidance without checking how this feature actually looks or works on the user's phone, including cases where it mentioned a settings path or icon that does not exist on the device."

Create a new annotation queue with all score configs: the original open_coding and pass_fail_assessment plus one boolean per failure category. Langfuse annotation queues can't be modified after creation, but scores on observations are preserved. Re-add the same 100 observations to the new queue and the previous notes and pass/fail scores will still be visible while annotators apply the category labels.

Step 4.2: Compute failure rates

The failure rate for a category is the percentage of traces where that category was marked true.

In Langfuse: Dashboards → Add Widget → Data source: Scores → Metric: Average → filter to the score name. Average value of a boolean score equals the failure rate. For a combined view: one bar chart grouped by score name.

Dad Tech Support results (illustrative, based on 19 labeled traces; complete all 100 before finalizing priorities):

impersonates_child                  58%  ████████████
identity_not_disclosed              42%  ████████
tone_persona_off                    42%  ████████
too_verbose                         32%  ██████
denied_scope                        16%  ███
missing_device_lookup               11%  ██
missing_clarifying_question         11%  ██ 
incomplete_resolution                5%  █

The identity cluster dominated. impersonates_child (58%) and identity_not_disclosed (42%) shared the same root cause: the persona instruction was miscalibrated. tone_persona_off (42%) was likely a downstream symptom. All three pointed at the same prompt fix.

Step 5: Decide what to do about each category

Work top-to-bottom by failure rate. For each category, ask in order:

1. Can we just fix it?

Fix first. Don't build an evaluator for something you can resolve in the prompt.

2. Is an evaluator worth building?

Not every remaining failure needs one:

• Is the failure rate high enough to matter?
• What's the business impact when it occurs?
• Will someone actually iterate on this evaluator?

3. What kind of evaluator?

Langfuse has a built-in online evaluation feature that runs LLM judges automatically on new traces. Check this before writing anything custom.

Dad Tech Support decisions:

What comes next

After one round of error analysis you have a prioritized list of things to fix and a set of categories to monitor.

1. Apply the prompt fixes. Use Langfuse prompt management to version and track changes.
2. Set up evaluators for categories that warrant them, starting with the highest-impact failure requiring judgment. See info on running LLM-as-a-judge
3. Re-run error analysis after the next significant change. Failure distributions shift. A prompt fix can resolve one category and reveal another. Run error analysis after prompt rewrites, model switches, new features, and production incidents.

Common mistakes