Microsoft researchers found that the first error in a failed agent run is rarely the actual cause, and propose a structured nine-category taxonomy to diagnose failures accurately.
Adapted from @undefinedKiMicrosoft researchers just found why your AI agent fails. The fix is stupidly simple and I have not seen anyone do it. They hand-labelled 170 failed agent runs. 62 hours of work, three annotators, near-perfect agreement on what went wrong in each. The finding: the first error in a trace is usually not what killed the run. Example: A web agent fails to download a PDF at step 3. Looks like the cause. But the orchestrator recovered, switched to another agent, carried on. The run actually died at step 33, when a file was saved to one path and then read from another that never existed. Thirty steps apart, and fixing the first changes nothing. So here is the question. At every error in the log, ask whether the agent recovered from it. Did it retry, switch tools, find another route, keep making progress? If yes, that error is noise the system already survived. Keep scanning. The first one it never came back from is your actual bug. Then name the failure type before you change anything, because the fix is different for each. Their nine categories condense to this: It skipped a step it was told to do. It invented a fact that was in no input. It called a tool with bad arguments. It misread what a tool returned. It misunderstood the goal. It lacked information nobody gave it. It was asked for something no tool can do. A guardrail blocked it. Or something was simply down. Bad arguments is a schema problem. Misread output is a parsing problem. Skipped step is a prompt problem. Treating all three the same is why agent debugging feels endless. Write the label down every time. After ten failed runs you have a distribution instead of a feeling. One last thing from their tests. A judge with no evidence in front of it labelled almost everything "did not follow instructions", because that is the easy answer when you cannot see what really happened. If that is your default diagnosis too, you are guessing.
