Measuring AI Agent Latency as a Distributed System

# Agents as distributed systems. 

The model sits inside a request path with queues, synchronization barriers, retries, caches, and external dependencies. I have found more latency in that path than in any single inference setting.

The completed run is the useful unit of measurement. A provider dashboard can tell me time to first token, tokens per second, input size, output size, and model processing time. The user waits for admission, authentication, state loading, planning, inference, retrieval, database queries, code execution, evaluation, persistence, and delivery. They experience the composition.

This is why I treat an agent as a small distributed system. I draw the dependency graph, instrument its boundaries, assign deadlines, control fan-out, and keep enough execution state to explain how a request finished. Faster models help when inference lies on the critical path. They have limited effect when the request is queued behind another run or blocked on a search API.

The best latency work I have seen follows a fairly plain sequence. Measure the deployed request. Find the barriers. Run independent work concurrently within a budget. Move deterministic operations out of the model loop. Reduce the state passed between model calls. Then load-test the result with the same quality checks used for the slower version.

That sequence also catches most of the failures: hidden serialization, unbounded fan-out, retries that multiply load, global mutable state, stale caches, oversized contexts, and one slow branch holding an otherwise complete answer.

## The completed run is the unit of measurement

I start one clock at the public request boundary and stop it when the result is durable and available to the caller. Every internal operation becomes a child span of the same run. I record its parent, operation type, dependency, queue delay, execution time, retry count, cache status, input and output size, and terminal state.

The parent relationship is necessary because a flat list of timings cannot show which operations overlapped. Adding every span can produce a duration longer than the request itself. The trace needs to reconstruct the schedule.

I keep three measurements separate:

- Time to first useful event — How long did the interface appear idle?

- Time to final result — How long did the requested work take?

- Sustainable throughput — How much work can the system finish while preserving its objectives?

Streaming affects the first measurement. Worker capacity affects the third. Concurrency within a run can shorten time to the final result while reducing system throughput if it overwhelms a shared provider. Any performance claim should say which of these moved and what happened to the others.

A simple agent loop is usually drawn like this:

> model -> tool -> model -> tool -> model -> result

I redraw it as a dependency graph. Some tool calls need values produced by an earlier call. Others happen later because someone wrote them later. Only the first kind is a data dependency.

For each edge, I record why the next operation is waiting:

- A data dependency means an input does not exist yet.

- A policy dependency means the system has chosen to wait.

- A resource dependency means the work is ready and capacity is unavailable.

- A presentation dependency means useful state exists and the interface has not exposed it.

- An accidental dependency comes from implementation order.

Policy, resource, presentation, and accidental waits give me a concrete place to investigate. A data dependency may disappear under a better plan, although adding async does not change what information the next operation needs.

Within a group of independent operations, the slowest required branch sets the wall time. Dependent stages accumulate. I model a run as a sequence of parallel waves:

> run time ~= queueing + sum(critical operation in each wave) + finalization

The critical path can shrink even when the system performs more aggregate work. That trade becomes useful when the additional work is safe to overlap and the shared infrastructure has room for it.

LLMCompiler uses a similar representation. A planner produces a dependency graph, a scheduler dispatches ready tasks, and an executor runs independent functions concurrently. Its benchmark results apply to its workloads, while the graph abstraction applies much more broadly. Kim et al., An LLM Compiler for Parallel Function Calling (https://arxiv.org/abs/2312.04511)

## Model turns are synchronization barriers

A conventional tool loop repeatedly transfers control across a network boundary:

> model decides
application parses
tool executes
application serializes
model reads
model decides again

One turn can include provider queueing, transport, inference, parsing, and another opportunity to fail. I review an agent plan by asking whether control needs to return to the model at each boundary.

Sorting, deduplication, arithmetic, schema validation, and repeated transformations usually belong in code. A scheduler can dispatch known tool dependencies. The application handles those steps directly; the model stays in the loop when new evidence can change the plan.

Programmatic tool calling pushes this further. The model emits a bounded program, a sandbox runs its control flow and tools, and the reduced result returns to the model. Intermediate records stay outside the conversation, which removes inference turns and repeated context processing. AWS, Implementing programmatic tool calling on Amazon Bedrock (https://aws.amazon.com/blogs/machine-learning/implementing-programmatic-tool-calling-on-amazon-bedrock/)

This often leaves a hybrid architecture. Deterministic stages run as a workflow, while the model chooses paths at the points that require judgment. Anthropic describes the same division in its distinction between workflows with predefined paths and agents that direct their own tool use. Anthropic, Building effective agents (https://www.anthropic.com/engineering/building-effective-agents)

Independent work should run concurrently when the merge is well defined. Searches over unrelated sources may qualify. So may evaluations of separate candidates under one rubric. Fetching a document and using its contents to write the next query does not.

I represent those dependencies outside the prompt so the scheduler can release a task as soon as its inputs exist. Its output goes through a deterministic reducer. This gives me a few scheduling patterns:

- Fan-out and gather dispatches independent branches and merges the results.

- Pipeline overlap starts downstream work on complete partitions while the remaining partitions continue.

- Hedged reads issue a duplicate after a delay when a safe dependency develops a long tail.

- Speculative reads begin a likely side-effect-free operation while the authoritative path is deciding.

- Incremental reduction filters or deduplicates partial results before the whole wave completes.

Each one has a cost. Hedging spends extra capacity, speculation wastes work when the prediction is wrong, and an incremental reducer can make the answer depend on arrival order. I use them only when the trace shows enough waiting time to justify the additional coordination.

Anthropic’s research system runs parallel work at two levels: a coordinator delegates separate research directions, and each worker can issue independent tools at once. The same account says parallel research consumes substantially more resources and fits poorly when workers need shared context or tightly ordered work. Anthropic, How we built our multi-agent research system (https://www.anthropic.com/engineering/multi-agent-research-system)

Adding agents without mapping their dependencies tends to add messages and inference calls. The scheduler produces the speedup.

## Concurrency spends shared capacity

An outer request may create several workers. A worker may issue several model calls, and a model response may request several tools. A modest product-level concurrency setting can turn into a large downstream burst.

I set limits at each boundary:

> per-user active runs
per-run workers
per-worker tool calls
per-tool concurrency
per-provider requests and tokens
global execution slots

The user limit provides fairness. The run limit contains a bad plan. Tool and provider limits protect dependencies from bursts. The global limit keeps the process inside its memory, connection, and worker budgets.

The queue behind each limit needs its own maximum size, ordering policy, deadline, and admission rule. A semaphore that protects the provider while requests wait past their useful deadline has moved the failure into the queue. I prefer an explicit rejection or deferral policy once the service runs out of capacity.

An agent also waits for the slowest required dependency. More fan-out raises the chance that one branch encounters a slow response, even when each dependency has a healthy average. Distributed services have dealt with this tail problem for years; agents inherit it and add nondeterministic work on top.

Each operation receives a deadline inherited from the run and a local timeout that fits inside the remaining budget. The retry policy depends on idempotency and error type. Optional work needs a fallback, and the scheduler cancels work that can no longer affect the answer.

Retries deserve particular care. Nested clients can retry independently until one failure becomes a burst of attempts at every layer. Immediate retries often arrive while the dependency is still unhealthy. I use bounded attempts, jittered backoff, and a retry budget shared by the run. A deterministic client error is not retried as if it were a transient server error.

For optional branches, partial completion may be acceptable. A required evidence request that times out should remain missing in the result. Rewriting it as a plausible model answer would improve apparent completion by changing the task.

Production tracing guides identify slow tools, excessive generation, memory growth, and serial execution as separate sources of latency. I want the trace to preserve that separation. AWS, Optimizing production agents with AgentCore Observability (https://aws.amazon.com/blogs/machine-learning/optimizing-production-agents-with-amazon-bedrock-agentcore-observability/)

## The data plane includes tools and context

Model inference is visible, metered, and easy to blame. Agents also wait for search, browsers, databases, storage, code sandboxes, and third-party APIs. From the caller’s perspective, all of that time belongs to the agent.

Bian et al. separate model API latency from web-environment latency and find workloads where either one dominates. Their SpecCache design overlaps predicted environment work with the authoritative reasoning path. Bian et al., What Limits Agentic Systems Efficiency? (https://arxiv.org/abs/2510.16276)

I profile each tool as a service. I want its latency distribution, queueing behavior, concurrency response, connection reuse, retry semantics, cache safety, and regional placement. I also check whether simultaneous identical reads can share one in-flight request and whether a slower dependency has a partial substitute.

An orchestration framework cannot compensate for an unindexed query or an HTTP client that opens a fresh connection for every call.

Caching helps at several layers. Intra-run memoization prevents one trajectory from repeating an identical read. In-flight coalescing lets concurrent identical reads share a future. Cross-run caching reuses a completed result under an explicit identity and freshness policy. I start with the narrowest scope that removes the repeated work.

A tool-result cache key needs every input that can change the correct answer:

> tool identity
normalized arguments
source and schema version
authorization scope
user or tenant boundary
freshness class

An omitted authorization scope can serve one user’s result to another. An omitted source version can make valid bytes describe an obsolete schema. The cache has to preserve the identity of the computation, not simply its output.

Prompt caching has a different contract. Stable instructions, tool definitions, rubrics, and shared documents should sit where the provider can reuse them. Dynamic request data should not invalidate that prefix without a reason. I record cache-hit tokens because a cache configuration without observed hits tells me very little. OpenAI, Model guidance, Google Cloud, Context caching (https://developers.openai.com/api/docs/guides/latest-model) (https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-context-caching)

Self-hosted inference permits reuse of encoded model state. Prompt Choreography explores a shared cache across calls in a language-model workflow. Its serving and training design differs from an ordinary API client, but the work identifies duplicated computation that appears in many multi-call systems. Bai and Eisner, Accelerating Language Model Workflows with Prompt Choreography (https://aclanthology.org/2026.tacl-1.13/)

Long context creates another data-plane cost. Every raw tool response left in history may be processed by later turns. The prompt grows, relevant state becomes harder to locate, and cache reuse becomes more fragile.

I treat a tool response as an interface. It returns the smallest structured evidence needed for the next decision and keeps a stable source identifier for deeper inspection. Large records are filtered or aggregated first. Durable state lives in typed storage instead of surviving only as conversation text. Old interaction history is summarized under a stated loss policy. When the tool registry is large, I expose the relevant subset for the current stage.

Generated tokens have a separate cost because the model produces them sequentially. A routing or extraction node should have a narrow output contract. OpenAI’s latency guidance treats shorter outputs, fewer requests, parallel execution, streaming, and deterministic substitutes as distinct techniques. OpenAI, Latency optimization (https://developers.openai.com/api/docs/guides/latency-optimization)

I am trying to preserve the smallest state that supports the next decision. Prompt length by itself is a poor objective.

## State and quality under concurrency

A prototype can keep the active run, progress, cancellation flag, or latest result in process-global variables. Serialization hides the bug. Under concurrent traffic, one request overwrites another, a cancellation reaches the wrong run, or a late callback publishes into a completed response.

Every run needs a stable identity and isolated state. An update succeeds only while the run still owns the relevant lease or version. The service persists the result before it announces completion, and late tool responses cannot reopen a terminal run.

Idempotency handles ambiguous submissions. A client may time out without knowing whether the server accepted its request. With an idempotency key, a retry returns the existing run. Effectful tools require their own keys because an inner action can be replayed even when the outer request was deduplicated.

Durable execution prevents a reconnect or worker restart from repeating completed work. It does not reduce the runtime of a successful attempt, although it can greatly reduce the time a user spends recovering from a failed one.

Quality needs the same run identity as latency. Nearly every speed technique can make the dashboard look better by weakening the job: call fewer tools, stop earlier, use stale state, drop a late result, return less evidence, or switch to a weaker model.

I define the quality contract before the optimization. The exact checks depend on the task:

- Execution — The run reached a valid terminal state

- Structure — Required fields parse and satisfy the schema

- Tool behavior — Required tools and parameters were correct

- Evidence — Claims retain support from permitted sources

- Outcome — The user-visible task was completed

- Safety — Effectful operations respected policy and authorization

The comparison uses the same input distribution and frozen protocol. For a probabilistic system, I prefer a predeclared non-inferiority threshold to a claim of perfect equivalence. When a model judges the result, a human-reviewed slice should test whether the judge shares the same blind spot as the optimized agent.

Reasoning effort and model choice belong inside this contract. A smaller routing model may be a good trade after it passes the routing evaluation. Applying it everywhere because it is faster has changed the inference hypothesis before measuring the effect.

## The interface can show real work

A long-running request should expose useful state before the final prose exists. During the tool and planning stages, I stream execution events:

> request accepted
plan created
retrieval branches active
evidence under review
final synthesis started
result persisted

These events correspond to state transitions. I avoid invented completion percentages because an agent may revise its plan and create more work.

The interface reconnects with a run identifier and distinguishes queued, running, waiting, completed, failed, cancelled, and completed-with-missing-evidence states. Cancellation propagates to pending tools and model requests where the dependency supports it.

Streaming reduces the silent wait. Durable job state allows a request to survive a disconnect. Neither changes the final completion time, so I report them separately from latency improvements. Vercel, The Agent Stack, Vercel, How to build scalable AI applications (https://vercel.com/blog/agent-stack) (https://vercel.com/blog/how-to-build-scalable-ai-applications)

Voice agents work under a tighter interaction budget, and their observability model is still useful for slower agents. Speech recognition, inference, tools, synthesis, and transport are timed independently. LiveKit, Understand and improve agent latency (https://livekit.com/blog/understand-and-improve-agent-latency)

## Match the architecture to the work

Multi-agent systems buy context isolation and parallel exploration. They pay for delegation, repeated inference, communication, aggregation, and additional error paths. The dependency structure decides whether that exchange is worthwhile.

- One tightly dependent reasoning path — One agent

- Known deterministic stages — Workflow

- Independent evidence branches — Parallel workers with one coordinator

- Repeated bounded transformation — Code or batch execution

- Environment-dependent planning — Agent loop

- Independent judgments under one rubric — Parallel evaluators with deterministic merge

Google Research tested several coordination architectures and found different results for parallel and sequential tasks. It also found that architecture changed how errors propagated between workers. Google Research, Towards a science of scaling agent systems (https://research.google/blog/towards-a-science-of-scaling-agent-systems-when-and-why-agent-systems-work/)

I require each proposed worker to own a separable task with a clear input, output, and merge rule. Specialization without that boundary adds an agent and leaves the original dependency intact.

Capacity follows the same reasoning. Registered user count says little about the work arriving at the system. I need the arrival process, run duration, internal fan-out, and shared resources.

Little’s Law provides the basic relationship:

> work in system = arrival rate * time in system

Shorter service time releases an execution slot sooner. Parallel work within the run may shorten service time while occupying more downstream slots. I measure the combined effect at the public boundary.

The capacity report describes an operating envelope: request arrival rate and burst shape, active run count, model and tool concurrency, queue depth, end-to-end latency, failures, quality outcomes, and resource headroom.

I use several terms that are easy to blur. Maximum observed concurrency is whatever happened during a test. Maximum validated concurrency is the highest tested level that met every objective. Sustainable concurrency needs a steady workload and operational headroom. The breaking point is where latency, quality, errors, or resource safety leaves the accepted envelope.

The load test calls the deployed route. Invoking an internal function skips authentication, network transport, runtime scheduling, state loading, persistence, and the proxy limits that users encounter.

I raise concurrency in steps and hold each level long enough to see queues and tails. The workload includes easy, ordinary, and adversarial requests in their expected proportions. I report cold, warm, cached, and uncached behavior separately.

At each step I record:

- accepted, rejected, completed, timed-out, and failed runs;

- queue time and execution time;

- end-to-end and stage-level latency distributions;

- provider throttling and retry volume;

- worker, connection-pool, memory, and database pressure;

- cache hits, misses, and coalesced requests;

- model and tool fan-out;

- quality outcomes;

- work that continued after cancellation.

The test stops when an objective fails. Testing beyond that point can describe the collapse mode, but I do it deliberately and protect the downstream services. Each submission also receives a unique run identity and input so a cache does not turn a concurrency test into repeated reads of one result.

## Failure modes I check first

Hidden serialization appears as a loop with an await inside it. The trace has no overlap even though the calls use independent inputs.

Model-mediated control flow asks the model to approve deterministic routing, filtering, counting, or formatting. Trivial decisions accumulate into inference barriers.

Unbounded fan-out makes one request fast and several simultaneous requests unstable. Local latency improves until provider throttling moves the wait into retries and queues.

A straggler join waits for every branch, including optional outputs that can no longer change the result.

Retry amplification occurs when nested clients each apply their own retry policy. One transient failure produces many attempts, sometimes after the caller’s deadline.

Context accretion leaves every raw tool response in history. Later turns repeatedly process evidence they no longer need.

A cache without full identity omits authorization, source version, or freshness. It returns a quick answer across a correctness or privacy boundary.

Global run state assumes one active request. Progress, cancellation, and final output attach to whichever run wrote the shared value most recently.

Streaming theatre emits generic activity while the backend is blocked. The first animation arrives quickly and the first useful information does not.

Average-only reporting hides the tail. A minority of requests can spend most of their time on stragglers while the dashboard remains healthy.

A quality-changing optimization calls fewer tools, searches fewer paths, drops results, or swaps models without a non-inferiority test.

Timeout inflation allows the request to run longer and reports the higher completion rate as a reliability improvement. The definition of failure moved; the work did not get faster.

I check these before touching the model because standard traces and load tests can usually confirm or reject them.

## How I approach an optimization pass

I trace the deployed request from admission to the persisted result, including parent-child relationships and queue time. Representative traces become dependency graphs with each wait labeled by its cause.

The first code changes remove accidental barriers. Independent reads run concurrently, reducers become deterministic, and the scheduler cancels work that cannot affect the answer. I then remove model round trips that perform fixed transformations or repeat a known plan.

The next pass controls data movement. Tool responses become smaller interfaces, stable prompt prefixes are arranged for reuse, and old state stops growing without a policy. Cache hits and misses enter the trace.

Concurrency controls follow: queue bounds, admission rules, deadlines, idempotency, and separate budgets for runs, tools, and providers. I repeat the deployed load test after each scheduling change because a faster isolated run can still damage throughput.

Real progress events, reconnection, and cancellation improve the user experience once the execution state is trustworthy.

Model changes come later. Reasoning effort, routing models, and provider service tiers can all help, but each one changes cost or inference behavior and needs an evaluation of its own. The trace may show that a model call dominates from the beginning, in which case I move that experiment forward. The order follows the evidence.

## What I don’t know

I do not know a universal architecture for agent latency. Research, coding, voice, transactional, and background agents have different dependency graphs and different costs for partial completion.

There is no context-length threshold I would carry unchanged across providers and models. Cache behavior, prefill performance, tool schemas, and the rest of the request path move it.

The right concurrency limit depends on decomposability, downstream quotas, burst shape, and how much additional work the system can spend to reduce wall time. A provider benchmark cannot settle this for a deployed application with its own network path and traffic.

An automated quality gate also establishes only what its protocol checks. A judge may share the failure introduced by the system it evaluates.

The frame is more stable than any particular setting. An agent is a distributed computation with probabilistic workers. Inference contributes latency alongside barriers, queues, data movement, retries, stragglers, and coordination. Throughput depends on how the run consumes shared capacity, and the optimization is valid only while the result keeps its original evidence and authority.

## Sources and related reading

These are the research papers, provider engineering posts, and field guides I found most useful for this topic. The surrounding literature continues to change.

1. OpenAI. Latency optimization. Output length, request count, parallelism, streaming, and deterministic alternatives. (https://developers.openai.com/api/docs/guides/latency-optimization)

1. OpenAI. Model guidance. Reasoning effort, prompt caching, tool design, and state management. (https://developers.openai.com/api/docs/guides/latest-model)

1. Anthropic. Building effective agents. Workflows, agents, routing, and parallelization. (https://www.anthropic.com/engineering/building-effective-agents)

1. Anthropic. How we built our multi-agent research system. Coordinator-worker execution, parallel research, evaluation, and production trade-offs. (https://www.anthropic.com/engineering/multi-agent-research-system)

1. Anthropic. Prompting best practices. Parallel tool use, reasoning effort, and avoiding unnecessary work. (https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prompt-templates-and-variables)

1. Google Research. Towards a science of scaling agent systems. Matching coordination architecture to task structure. (https://research.google/blog/towards-a-science-of-scaling-agent-systems-when-and-why-agent-systems-work/)

1. Kim et al. An LLM Compiler for Parallel Function Calling. Dependency-aware planning and concurrent tool execution. (https://arxiv.org/abs/2312.04511)

1. Bian et al. What Limits Agentic Systems Efficiency?. Model and environment latency, caching, and speculation. (https://arxiv.org/abs/2510.16276)

1. Bai and Eisner. Accelerating Language Model Workflows with Prompt Choreography. Reuse of encoded state across model workflows. (https://aclanthology.org/2026.tacl-1.13/)

1. Zhu et al. Divide-Then-Aggregate: An Efficient Tool Learning Method via Parallel Tool Invocation. Graph decomposition and parallel tool execution. (https://aclanthology.org/2025.acl-long.1401/)

1. Asynchronous LLM Function Calling. Non-blocking function execution and interrupt semantics. (https://arxiv.org/abs/2412.07017)

1. AWS. Implementing programmatic tool calling on Amazon Bedrock. Reducing model round trips with bounded code execution. (https://aws.amazon.com/blogs/machine-learning/implementing-programmatic-tool-calling-on-amazon-bedrock/)

1. AWS. Optimizing production agents with AgentCore Observability. Tool, memory, token, and scheduling bottlenecks. (https://aws.amazon.com/blogs/machine-learning/optimizing-production-agents-with-amazon-bedrock-agentcore-observability/)

1. LangChain. Multi-agent systems. Coordination patterns and their model-call, token, and parallelism trade-offs. (https://docs.langchain.com/oss/python/langchain/multi-agent)

1. Vercel. The Agent Stack. Model access, durable execution, and long-running agent infrastructure. (https://vercel.com/blog/agent-stack)

1. Vercel. How to build scalable AI applications. Streaming, backpressure, caching, and application scaling. (https://vercel.com/blog/how-to-build-scalable-ai-applications)

1. LiveKit. Understand and improve agent latency. Stage-level observability for real-time agents. (https://livekit.com/blog/understand-and-improve-agent-latency)

1. General Compute. Evaluating agent performance: latency as a first-class metric. End-to-end and per-step agent measurements. (https://www.generalcompute.com/blog/evaluating-agent-performance-latency-as-a-first-class-metric)