How to set up a user feedback loop for your application

Once traces are coming in, capturing user feedback is a great next step. Your users judge every response through what they click, edit, retry, or say. Capturing that on your traces gives you a quality signal grounded in real usage, where each score points at a trace you can open and learn from.

A lot of teams ask "I have traces coming in, how do I now get started with evals?". Often, these teams are better off starting with user feedback first.

This feedback gathering step generally has a very high ROI, in the AI Engineering process. We often write about this process in the Langfuse Academy, where we walk through the AI engineering loop in more detail. Gathering user feedback happens in the monitoring stage highlighted below.

Prerequisites

You already have traces coming into Langfuse. See tracing if you have not set this up yet.

Walkthrough

Choose your feedback signals

Feedback comes in two forms. Explicit feedback is a rating the user gives on purpose, like a thumbs up or down or a star rating: unambiguous, but rare and skewed toward unhappy users. Implicit feedback is derived from what users do, like retrying a query or editing a draft: you have data on every trace, but it needs interpretation.

The best signals depend heavily on your use case. Some examples as inspiration:

Tips:

• signals combine well, no need to implement only one
• capture free-form text wherever it fits (can be as a comment on the score), as much of this analysis is now done by agents, and they work better with more context
• the academy examples show how signals are chosen for a specific application and change over its lifecycle

Capture the signals as scores

Every signal ends up as a score on the trace that produced the output. A score has a name, a value with a data type (boolean, numeric, categorical or free text), and an optional comment. Evaluators also store their results as scores, so your user feedback can land in the same filters, dashboards, and analytics.

Act on the feedback

Monitor how quality is trending

Score analytics and custom dashboards chart how each signal moves over time. One caveat: not every signal is a clean quality metric, e.g. thumbs feedback skews toward unhappy users.

Surface interesting traces

Filter traces by score, for example response_rating = 0, to get a concrete list of traces to investigate. Error analysis is a structured way to read and cluster them, and an annotation queue brings in more reviewers.

Use it as input for structured improvement

Add the failing cases to a dataset, test a fix with an experiment before shipping, and turn a recurring failure mode into an automated evaluator that scores every production trace from then on.

Much of this can also be handed to an agent, see Let an agent act on the feedback below.

Let an agent act on the feedback

Langfuse is built for agent access through the Langfuse CLI, the Agent Skill, and the MCP Server: point one at your project and it can interpret and act on the behavior that gets surfaced via user feedback. An example task:

Fetch the traces from the last 7 days with a response_rating score of 0.
Read the comments and outputs, cluster the failures into categories,
and propose a prompt change for the two biggest ones.

Some reading material as inspiration:

• Using Agent Skills to Automatically Improve your Prompts
• AI is eating the AI engineering loop