How traces and spans provide structured visibility into every operation of a RAG pipeline for debugging, cost analysis, and quality monitoring.
Adapted from @_avichawlaLayers of observability in AI systems, explained visually: If an LLM app is serving real users, its input and output are not enough to debug it. Consider a RAG pipeline where a query passes through embedding, retrieval, context assembly, and generation. Every operation adds latency, may call a paid API, and can fail while still producing a valid-looking response. Traces and spans provide visibility. - A trace records the full path of one request. The Trace column runs from query to response. - A span records one operation within that trace. The colored boxes are spans. Each span captures: > Query span The input, timestamp, session identifier, and request metadata. > Embedding span The model, input size, latency, retries, and rate-limit errors. > Retrieval span The retrieved chunks, document IDs, relevance scores, filters, top-k value, and latency. Many RAG failures originate here. Without these fields, there is no evidence that retrieval selected the wrong documents. > Context span The context assembled from retrieved chunks, instructions, and conversation history. This catches truncated documents, duplicated chunks, missing citations, and prompts exceeding the token budget. > Generation span The model, token counts, time to first token, total latency, finish reason, retries, and estimated cost. With these details, a bad response can now be traced to retrieval, context assembly, or generation. To use this in practice, Opik already implements this observability infrastructure for LLM apps and is open source. It captures traces and spans across LLM calls, retrieval steps, and tool executions, with latency, token usage, and cost attached to each operation. GitHub repo: https://t.co/vahjkkfJCt (don't forget to star it ⭐) In Opik, every operation belonging to one request carries the same Trace ID. If the app processes 1,000 requests, it creates 1,000 traces, each containing its own spans. This makes cost analysis more useful. Instead of aggregate spend, teams can identify the model calls, retries, or oversized prompts responsible. Over time, changes in retrieval scores, embedding latency, or context size become visible before they turn into broader quality problems. That said, observability is one of eight areas I would learn for building production LLM systems. I covered all eight in the 2026 LLM Engineering Roadmap, with free and open-source resources for each one. Read it below.
