How to instrument AI applications to capture implicit user behavior signals as scalable, low-cost quality evaluations.
Adapted from @lotte_verheyden# Your users are already evaluating your AI I have traces, now I need evals. We do these "Welcome to Langfuse" calls at Langfuse, and the sentence above is a very common way for users to open the conversation. At this point users have 2 choices: - set up online evaluators that measure specific aspects of a trace (https://langfuse.com/academy/evaluate) - capture signals of judgment from your users (https://langfuse.com/docs/observability/features/user-feedback) This article talks about the latter. It's also what we recommend you start with. Unlike traditional online evaluation, capturing signals does not directly evaluate the quality of the agent's work. Instead, it captures things a user did or said that indicate whether the user perceives the agent's work as good or bad: a thumbs down, a retry, a request for a human, etc. > Signals measure perceived quality at the moment of use. ## # Why this is important This is a cheap and scalable way to get quality judgments on production traffic, compared to other methods such as experts annotating traces (human time) or online LLM-as-a-judge evaluators (LLM cost). Your users judge every output anyway, through what they do next, so you just need to instrument your application to catch these signals. While through traditional evaluation you encode what you believe is important, user signals will also surface behaviors you didn't anticipate but might be worth taking a look at. This makes them especially valuable in staying on top of how users experience your agent's work. # Four kinds of signals So what kinds of signals are we talking about? We can divide user feedback signals into four kinds: explicit ratings, behavioral signals, conversation signals, and outcome signals. ## Explicit ratings Explicit ratings are the only signals where the user knowingly grades the output, which makes them unambiguous, but also rare. In Meta's production assistant, about 0.1% of model messages receive a positive emoji reaction, and the ratings that do arrive are skewed toward users with strong reactions. (https://arxiv.org/abs/2505.14946) (https://langfuse.com/academy/monitoring/capturing-signals#biases) ## Behavioral signals Behavioral signals are actions users take on the output: accepting, copying, editing, regenerating, skipping, abandoning. They occur inside the normal workflow, so coverage is high: on a copilot-style product, every suggestion produces one. ## Conversation signals In a conversational product, the user's next message often tells you what they thought of the agent's output: a rephrased question, a correction, a request for a human, or a thank you all say something about the quality of the output they received. > Separate steering from correction. In this music DJ example, "play something calmer" is normal use of the feature and "I said calmer" is an indication the agent didn't deliver what the user wanted; only the latter is an actual failure. (https://langfuse.com/academy/examples/music-streaming-dj) Amazon used rephrase and follow-up patterns to curate training data for Alexa's language understanding from live traffic. Microsoft's SPUR goes a step further: an LLM learns satisfaction and dissatisfaction patterns from the small slice of thumbs-labeled conversations, condenses them into a rubric, and scores the unlabeled majority with it. (https://arxiv.org/abs/2010.12251) (https://arxiv.org/abs/2403.12388) ## Outcome signals Outcome signals tell you what the environment did with the output: a draft was sent unedited, a ticket was closed, etc. They are the strongest kind because they are grounded in the environment rather than in someone's opinion. They are also often the most complex to implement. # How to interpret signals Once relevant signals are implemented, you can look at your traces as labeled for that signal: labeled negative, labeled positive, or unlabeled. > Which polarity is dense depends on the channel. Thumbs may skew positive while conversation signals tend to skew negative, for example. A negative is close to a confirmed bad experience. Users rarely retry or correct an answer they liked. Each trace with a negative user signal is worth opening, and negatives are the natural entry points for error analysis. (https://langfuse.com/academy/monitoring/error-analysis) A positive signal shows acceptance, but not necessarily quality. Read positives as a trend, and never make one the optimization target on its own. (https://langfuse.com/academy/monitoring/capturing-signals#which-signal) ## Who labels is not random Signals can be very useful, but you should also watch out for inherent biases. Often, you won't get a signal from the entire population of users. You can split these kinds of biases into: - Self-selection. People rate what they feel strongly about. When Yahoo! Music surveyed its listeners, 65% said their opinion of a song decides whether they rate it at all, and songs they loved or hated were rated far more often than songs they felt neutral about. (https://arxiv.org/abs/1206.5267) - Survivorship. All your signals are from users who stayed long enough to take the action you're tracking. Keep in mind that it's likely not the full set of users that started an interaction. - Exposure and interface. Users react only to what the system shows, in the form it is shown. For example, search users will keep clicking top-ranked results disproportionately regardless of whether they are the most relevant for their use case. (https://www.cs.cornell.edu/~tj/publications/joachims_etal_05a.pdf) - Cohorts. Signal rates vary across users, languages, and time of day without the quality of the output changing. ## Which signal should you optimize for? Signals are slightly different from explicit metrics and quality evaluators. Don't replace your goal metrics purely with signals. As for what to optimize for, prefer signals closest to what the product is for. Otherwise you might start optimizing for behavior you don't want. (https://langfuse.com/academy/evaluate/choosing-what-to-evaluate) Two examples of this: - YouTube ranked recommendations by clicks and got clickbait; ranking by expected watch time fixed it, because watch time is closer to what the product is for. (https://dl.acm.org/doi/10.1145/2959100.2959190) - When OpenAI added a reward signal built from ChatGPT thumbs data, the model turned sycophantic: user feedback favored agreeable answers and weakened the signal that had been holding sycophancy in check, while offline evaluations and A/B tests looked fine and only expert testers noticed the model felt off. (https://openai.com/index/expanding-on-sycophancy/) # What comes next Once signals land as scores on your traces, put them to work in four places: - Set up alerts on the signals that mark a bad experience, so that a spike in them reaches you right away. (https://langfuse.com/docs/observability/features/alerts) - Create filtered views of negatively labeled traces, so that you can go through them frequently. - Add graphs of signal rates to your dashboards to track how they develop over time. (https://langfuse.com/docs/metrics/features/custom-dashboards) - When reviewing flagged traces, use them: run error analysis on them and add them to datasets. (https://langfuse.com/academy/monitoring/error-analysis) (https://langfuse.com/academy/datasets) ## This is a page from the Langfuse Academy. Explore more on langfuse.com/academy. (https://langfuse.com/academy)
