Lessons from a $1,600 LLM Eval Gone Wrong

# I wasted almost $1,000 running this eval. Here’s what I learned.

When Kimi K3 launched in late July, I originally built an eval that compared Kimi K3 to Sonnet 5. About an hour after I published the blog to Braintrust, we had to pull it because other inference providers, who apparently have it out for me, had launched support for Kimi K3.

They were marketing how fast and reliable Kimi was on their stacks, and my original framing and numbers were out of date because my eval had run Kimi K3 on Moonshot. This meant that it was possible that Kimi K3 could have performed better if it ran on another serving stack, or that Moonshot was overloaded with traffic during the weekend I ran my eval. That possibility added too much uncertainty to the numbers in my original eval for us to feel good publishing them.

I did still post the article on my personal X, and you can find it here. (https://x.com/compose/articles/edit/2085806036557111296)

That was the first time I even thought about the serving stack behind an open-weight model being a factor in an eval, because I had only ever evaled closed models like GPT, Claude, and Gemini, where you don’t need to worry about the infrastructure serving it because there is only one option.

So instead of comparing the models, I compared the serving stacks: Moonshot, Fireworks, and Baseten, all serving Kimi K3.

This decision led me down a month-long process that made me question my own sanity, but after finally closing the book on it, I want to talk about what I learned in terms of best practices of what to do (and NOT to do) when running an eval.

Oh, and if you want to read the official blog (that made me question my own sanity), which contains the methodology and results from the eval, you can find it here. (https://www.braintrust.dev/evals/kimi-k3-serving-stacks)

→ How this eval got expensive

Each run of the eval was long and expensive. I had 20 tasks, where each task was creating an HTML page based on a screenshot using the Figma MCP server. I ran each task three times per serving stack, so each provider had 60 intended runs.

On average, one Fireworks run took 13.8 minutes and one Moonshot run took 16.2 minutes. Since the final provider runs used concurrency 1, a full 60-run pass was about 13 hours and 49 minutes of agent runtime for Fireworks, or 16 hours and 14 minutes for Moonshot.

In total, I spent $542.60 on Baseten, $661.82 on Fireworks, and $400 on Moonshot: $1,604.42 overall.

I went back through every active and archived experiment in the serving-stack project, including smoke tests and incomplete attempts.

When I say “wasted money” here, I mean money spent on runs that did not make it into the final retained analysis. I compared the runs that made it into the final analysis with the full set of recorded task runs in order to find what percentage of that total amount spent was “wasted money.”

\small
\begin{array}{l|r|r|r|r|r}
\textbf{Serving stack} &
\begin{array}{c}\textbf{All recorded}\\\textbf{task runs}\end{array} &
\begin{array}{c}\textbf{Retained in}\\\textbf{final analysis}\end{array} &
\begin{array}{c}\textbf{Runs not}\\\textbf{retained}\end{array} &
\textbf{Total cost} &
\begin{array}{c}\textbf{Estimated spend on}\\\textbf{runs not retained}\end{array}
\\ \hline
\text{Baseten} & 24 & 7 & 17 & \$542.60 & \$384.34 \\
\text{Fireworks} & 235 & 60 & 175 & \$661.82 & \$492.84 \\
\text{Moonshot} & 85 & 60 & 25 & \$400 & \$117.65 \\
\hline
\textbf{Total} & \mathbf{344} & \mathbf{127} & \mathbf{217} &
\mathbf{\$1{,}604.42} & \mathbf{\$994.83}
\end{array}

So of the 344 recorded task runs I did across the project, only 127 made it into the final analysis. A very rough estimate is that I spent about $995, or around 63% of the total, on runs I did not ultimately use.

Obviously, this is an estimate. Some smoke tests were necessary to learn what was broken, and some failed runs may have cost more or less than a typical run. But the moral of the story is that I spent a lot of money learning things I should have learned for much cheaper.

→ The lessons I learned

1. Run a lot of smoke tests before a full run.

My biggest mistake was doing too many iterations of a full eval run just to find silly mistakes in how I set up my eval. For example, I had to add a proxy to pin the temperature to 1, because Moonshot required it for Kimi K3. I had to add a browser User-Agent to get Fireworks requests through Cloudflare. And I had to change my timeout cleanup so that killing an agent run would also kill its MCP child processes.

These are things I could have discovered by running smoke tests, but instead I wasted hundreds of dollars.

I had never run such an expensive eval before, so I got punished severely for my sloppiness here.

1. Validate rate limits and account tiers before a full run.

On my first full Baseten run, the eval was ruined because I quickly hit a rate limit. Baseten returned 653 HTTP 429 errors (meaning “too many requests”) and completed only 7 of the 60 intended runs.

Baseten’s documentation says that a 429 can be caused by either requests per minute or tokens per minute. Its Basic verified and Pro tiers list a limit of 120 RPM and 500,000 TPM. Here are the docs.

The problem is that I still can’t prove which exact limit I hit because I did not retain the rate-limit headers or log request-level token usage. The request rate was below the documented RPM limit, so the more likely explanation is token throughput.

I had chosen a really token-intensive task. Each agent request included a growing conversation, tool schemas, screenshots, and prior tool results. So this was not just a serving-stack problem — it was also a task-selection problem.

This created something similar to a “floor effect,” where the task hit an operational floor before I could use it to distinguish quality between providers. (https://docs.baseten.co/inference/model-apis/pricing-and-limits)

1. Measure latency more than once.

Another best practice is that when benchmarking latency, it’s not enough to run the benchmark once. It’s better to run it on different days, at different times of the day, and to interleave providers instead of running one provider’s entire batch before the next.

This is because serving stacks might have different traffic, queueing, or batching behavior at different times.

Shoutout to my coworker Izzy, who called this out during a deep peer review of my research.

1. Delete failed runs.

Because of certain runs I had to stop halfway through or that failed due to tier limits, I had a couple of half-finished experiments in Braintrust. They were useful for a bit as debugging evidence, but cluttered up the project and confused later analysis.

When I asked an agent to use the Braintrust CLI or MCP to analyze the results, it would sometimes accidentally lump the half-finished runs into the analysis. Next time, I would archive or clearly tag invalid runs so it was clear which runs were eligible for analysis.

1. Concurrency changes the workload.

I believe we also ran into throughput issues when trying to run multiple agent tasks at once. With an agentic eval, each run can make many long requests with growing context, so increasing concurrency can create a much bigger token-throughput spike than you expect.

The final provider runs used concurrency 1, since I was already hitting token rate limits at concurrency 2 with Fireworks and Moonshot.

1. Control your eval environment.

Halfway through, I switched from Paper MCP, which was local to my laptop, to Figma MCP, which was hosted in the cloud. Paper MCP was a mistake because it was affected by the apps I had open, my machine load, and how many agents I was running at once.

That made it a much better setup for this serving-stack eval, but it’s something I should have thought about in the design of the task before I even ran my first smoke test.

1. Don’t jump straight from a smoke test to three trials.

I went straight from a smoke test to a full run of 20 tasks with three trials per provider. That turns a 20-task eval into 60 runs per provider before I even knew whether the setup worked at scale or whether there was a visible difference between the providers.

A better sequence would have been: run the smoke test, then do a thin full pass with one trial per task. At that point, I could have looked at the cost and early quality results and built a gut understanding of whether there was actually a difference worth measuring.

Though I’m not happy that the folks over at Baseten, Moonshot, and Fireworks got my money instead of me being able to go on an expensive shopping haul or pocket half an iPhone Duo, these are hard-earned lessons that will only improve my eval experience going forward. Hopefully you can learn from my mistakes as well 🙂