pass^k metric for measuring agent reliability

# Run your agent 8 times before you ship it 

# I found out my "90% accurate" agent was closer to 30%. Here's the metric that exposed it.

> TL;DRYour agent's success rate is a lie because you measured it once. τ-bench's pass^k metric measures whether it succeeds on all k tries, and the drop is brutal. GPT-4o: 61% on one attempt, under 25% across eight. I ran the same test on my own agent and it changed what I shipped. Tools and commands at the bottom.

I had a support triage agent sitting at what I was calling 90% accuracy. Forty-two test tickets, thirty-eight handled right. I put that number in a deck. I said it out loud in a meeting.

Then I read the τ-bench paper and rewrote the deck.

Here's the thing that got me. τ-bench introduced a metric called pass^k. Not pass@k, which most coding benchmarks use and which asks "did it succeed at least once in k tries?" That's an optimistic metric. It's great for code generation, where you can run tests and pick the winner.

pass^k asks the opposite question. Did it succeed on every single one of k tries?

For anything touching a customer or a database, that's the only question that matters. You don't get to run the refund eight times and keep the good one.

The numbers from the original paper:

Model pass^1 (retail) pass^8 (retail)     GPT-4o 61% under 25%   Claude 3.5 Sonnet (Oct 2024) 69.2% —   GPT-4o (airline) 42.0% —

Source: τ-bench, Yao et al.. The original board is frozen at late-2024 models; newer work lives on tau2-bench. (https://arxiv.org/pdf/2406.12045) (https://github.com/sierra-research/tau2-bench)

A model that looks like it works two times out of three falls under a quarter when you ask it to be consistent. And it's not just customer service. CORE-Bench found their agent at 22.2% pass^1 dropping to 8.9% at pass^3 on hard tasks. Same shape everywhere.

> Nobody's agent is as good as their demo. Not because they're lying. Because they ran it once.

## So I ran mine eight times

Same forty-two tickets. Eight runs each. Then I counted only the tickets that got handled correctly all eight times.

<!-- REPLACE THESE WITH YOUR OWN RUN — do not publish my numbers -->    Metric Result     pass^1 (what I'd been reporting) 90%   pass^3 61%   pass^8 33%

Thirty-three percent. That was the real number.

And the shape of the decay told me more than the value did. My failures weren't spread evenly across all forty-two tickets. They clustered. Eleven tickets failed almost every time and the rest were basically solid. That's a completely different engineering problem than "the model is flaky," and I'd never have seen it from a single run.

This is the part I'd push on if you take one thing away. Read the curve, not the number. If your pass^k decays gently, you have a consistency problem across the board and the fix is probably temperature, prompting, or model choice. If it falls off a cliff, you have a subset of tasks that are broken in a specific way, and the fix is in your code.

Mine was a cliff.

## What was in the cliff

Three things, and each one had an existing body of research that explained it better than I could have.

Context rot did most of the damage

The failing tickets were the long ones. Multi-turn threads where the agent had accumulated history.

Chroma's Context Rot study tested 18 frontier models — GPT-4.1, Claude 4, Gemini 2.5, Qwen3 — holding task difficulty constant and varying only input length. Every single model degraded. All eighteen. And it starts well before you're near the window limit, which is what wrecked my mental model. A million-token window is not a million usable tokens. (https://www.trychroma.com/research/context-rot)

Two findings hit my exact setup:

1. Distractors get much worse in long context. Similar-but-wrong information hurts far more than random noise. I was retrieving the five most similar past tickets on every turn. Similar tickets are, by definition, ideal distractors. I was manufacturing my own failure mode.

1. Coherent documents performed worse than shuffled ones. Every model, consistently. Nobody's fully explained why. I mention it because it should make you distrust your instincts about what "clean context" looks like.

Fix: capped tool output at 1,500 tokens, dropped retrieval from five documents to two, compacted history every six turns with a cheaper model, and hard-pinned the original task statement so it never gets compacted away.

pass^8 went from 33% to the high fifties on that change alone.

The agent had more access than I did

This one I found by accident, and it's the reason I now run a permissions audit before a benchmark.

Simon Willison calls it the lethal trifecta: privileged access, untrusted input, and an outbound channel. Any system with all three can be turned against itself with nothing but text.

My agent had database credentials. It read tickets written by strangers. It could post back into the thread. All three.

UpGuard has six documented cases and the pattern is identical every time. Invariant Labs showed a single malicious GitHub issue, filed from a free account, pulling private repo contents out of a developer's machine. A team on Cursor with Supabase service_role creds — a choice made for dev speed, which we've all made — got a support ticket containing instructions, and their agent queried the integration tokens table and posted the results back into the thread. (https://www.upguard.com/blog/mcp-security-incidents)

The fix isn't making the model resistant. It's removing what it can be manipulated into. If it reads untrusted input, no outbound channel. If it needs one, a human approves the payload. Permissions went from SELECT on the schema to four named columns.

OWASP's Top 10 for Agentic Applications lists this as ASI01, Agent Goal Hijack. Number one on the list. It's a ninety-minute read and it's the best ninety minutes available right now. (https://blog.cyberdesserts.com/ai-agent-security-risks/)

Running it eight times cost me real money

Obvious in hindsight. Eight runs on forty-two tickets is 336 agent executions, and agents resend their whole accumulated context at every step, so by step twenty you've paid for turn one twenty times.

Gartner's 2026 figure: agents burn 5 to 30× more tokens per task than a chatbot. Splunk's breakdown has the stat that actually landed for me — EY tracked one customer service interaction going from $0.04 to $1.20 over three years. Thirty times more expensive, while per-token prices were falling. (https://www.splunk.com/en_us/blog/observability/why-most-projects-still-die-before-production.html)

Uber's CTO said in April they'd burned the company's entire annual AI coding budget in four months.

So: run your pass^k on a subset first. Ten tasks, eight trials. Get the shape, then scale up. And alert on cost per task, not total spend — total spend is a lagging indicator, cost per task catches a runaway loop on its first occurrence.

## The five tools I actually run now

Tool What it's for When to reach for it     tau2-bench pass^k harness with simulated users + policy scoring Building your own domain eval   Langfuse Open-source tracing, self-hostable You want traces without a vendor   Braintrust Evals wired into CI, PR comments per run You want regressions caught before merge   Arize Phoenix OpenTelemetry/OpenInference tracing You already run OTel   E2B / Modal Sandboxed execution, per-run isolation Your agent writes and runs code (https://github.com/sierra-research/tau2-bench) (https://langfuse.com/) (https://braintrust.dev/) (https://phoenix.arize.com/) (https://e2b.dev/) (https://modal.com/)

A note on the harness: τ-bench scores are famously sensitive to setup. The repo shipped a v1.0.1 grading fix in July 2026 that re-scored the banking domain, and results from before that aren't comparable. Different user simulators and tool schemas produce different numbers for the same model. Which is fine — you're not trying to beat a leaderboard. You're measuring your own agent against itself over time. (https://github.com/sierra-research/tau2-bench)

Doing it yourself, in about an afternoon

1. Pick 20 real production inputs. Not synthetic. Ten should be ones that previously failed.

1. Run each 8 times. Fixed seed where you can, same model version, same tools.

1. Count only the ones that pass all 8. That's your pass^8. Write it down before you look at it, so you can't talk yourself into a better framing.

1. Plot the decay. pass^1, pass^3, pass^8. Cliff or slope?

1. Cluster your failures. If a small set of tasks fails repeatedly, read those traces line by line. That's where your bug lives.

1. Change one thing. Re-run. Only one. You cannot attribute a delta across two changes.

Twenty cases is enough. I wasted weeks planning a two-hundred-case eval suite I never built. Twenty bad evals beat zero good intentions.

## Where I landed

There's a survey from Pulumi of 510 engineers: 81% let agents touch production, but only 19% let them do it autonomously. Everyone else gates it behind a human. (https://www.pulumi.com/state-of-agentic-infrastructure/)

I used to read that as the industry underdelivering. Now I think it's just what shipping honestly looks like, and the people claiming otherwise haven't measured pass^8 yet.

My agent still runs. pass^8 is 71%. It can't write to a thread without approval, it sees four columns instead of a schema, and it has a step ceiling enforced by the runtime rather than requested politely in a prompt.

It's less impressive than the version I demoed.

It's the first one I'd actually put my name on.

If you run pass^k on yours, I want to see the curve. Especially if it's a cliff.