Agent Harness Experiments: Six Candidates, No Promotions

# Open the Harness. Learn What Makes an Agent Work.

A model can write excellent code and still be a frustrating coding agent.

It can find the right file, make a change, pass a few tests, and miss the actual request. It can recover from a broken tool call and forget what it was trying to finish. It can spend a thousand tokens reading something that changes none of its decisions.

If you want to understand why, open the harness.

The harness is everything around the model that turns a response into work: the instructions, tools, observations, memory, execution environment, and rules for continuing or stopping. Each part influences what the agent can notice and what it does next.

We recently spent almost two days examining and changing those parts in Agent Zero. We reviewed Terminal-Bench 2 and DeepSWE trajectories, then ran a separate set of local Terminal-Bench 4 experiments using our Revolve (https://github.com/agent0ai/revolve) workflow. We tested six candidate changes, kept detailed records, and fixed several concrete defects along the way.

None of the six candidates earned promotion into the general harness. That result and the lessons behind it deserve to be shared.

## An agent can have more than one shape

Start with a model choosing a tool, reading the result, and choosing again. That is one useful workflow.

Now give it a plan that persists between steps. Or make it localize the relevant code before proposing a patch. Or let a coordinator divide independent work among specialists and combine their findings. The model might stay the same while the system around it changes substantially.

A direct loop can adapt after every observation. A plan gives dependent steps a shared direction, but adds state to maintain. A localization-first workflow narrows the code under consideration, but can miss a dependency outside that boundary. Delegation separates independent investigations, but the coordinator still has to reconcile the answers.

Each shape changes where decisions happen, what context reaches them, and how a mistake propagates.

These are conceptual workflow patterns. Our measured experiments below changed narrower parts of the harness, rather than benchmarking these four complete architectures.

A useful harness lets you explore those choices. Agent Zero gives you several places to do it:

- Prompts shape how the agent approaches a problem. They are readable files, with profile-specific overrides.

- Tools and plugins shape the actions it can take and the feedback it receives. Core capabilities use the plugin system too.

- Memory and context shape what it can bring forward from earlier work.

- Profiles and delegation shape how responsibilities are divided.

- Projects and scoped settings let an experiment apply to a particular workspace or agent instead of changing everyone's defaults.

You can begin with a small prompt override. When the evidence points to a tool problem, you can inspect the tool. When the problem is what reaches the model, you can inspect that boundary too.

That makes the framework a place to learn through real interventions. A hypothesis can become a file change, a trace, and a measurable result.

## Six reasonable ideas, six reasons to measure

Agent Zero’s prompts are deliberately compact. We wanted to preserve that economy while improving behavior. Adding a sentence is easy; showing that it improves completed work is harder.

We began with ten CPU-only tasks in our local Terminal-Bench 4 setup. The main model was GLM-5.3-flash.

We used five tasks for development and reserved five from detailed failure analysis while selecting candidates. The first three candidates changed the solving guidance:

1. Organization. Make the connection between the requested behavior, authoritative inputs, relevant code, and deliverables more explicit.

1. Navigation. Ask for the smallest observation that can change the next decision, rather than broad searches and repeated reading.

1. Research momentum. Identify an unresolved assumption, choose an action that distinguishes explanations, then update the approach from the evidence.

All three sound sensible. Sensible is where an experiment begins.

Consider a five-task comparison. The baseline completes three tasks using 438,000 output tokens across the five attempts. Organization completes one using 562,000; navigation completes none using 491,000; research momentum completes one using 548,000.

In this example, every candidate uses more output tokens while completing fewer tasks. The decision is to retain the baseline. A reasonable-sounding instruction has to improve the work before it earns a place in the default harness.

Our actual experiments also produced no candidate that earned promotion; their measured results remain separate from this illustration.

We then tried three narrower changes: seed-memory recall, clearer Python-runtime guidance, and a factual remaining-time cue.

The recall candidate passed one additional partial check, but neither arm actually recalled memory. We could not credit the retrieval change for that difference.

The deadline candidate finished before the limit while its control timed out. Both still failed the task, passing the same one of five checks. One pair, with shared-host activity during the control, was insufficient to establish a benefit.

The useful discipline was keeping plausible changes out of the default harness until they earn their place.

## A successful-looking mistake

In the DeepSWE traces, we found implementations of incremental processing that collected the entire stream before parsing it.

The code could produce the expected eventual output. Existing tests could pass. The requested behavior, yielding incrementally, was still missing. This is an easy mistake to overlook if you measure progress by activity: files edited, commands run, tests passed.

A better question is: what observation would distinguish the behavior we requested from the easier behavior we accidentally built?

For streaming, that might mean checking whether the first result arrives before the full input is available. A large test count is less informative if none of those tests makes that distinction.

That lesson applies beyond verification. It changes how an agent reads a task, chooses an implementation, interprets feedback, and decides that it is done.

## A tool error is not one kind of problem

The traces also showed something worth resisting: treating every failed operation as a model-formatting problem.

An invalid call envelope, an unsupported patch operation, stale patch context, and a shell command that fails after execution need different recoveries. Context Doctor helps repair many serialized-call errors. It cannot make an unsupported operation exist or turn a wrong algorithm into a correct one.

In the DeepSWE operation logs we examined, edit failures appeared in 27 of 31 covered trials, including eight that eventually passed. Errors were common. The response to them mattered.

One concrete defect we reproduced was entirely below the prompt layer. An editor patch replaced a file with a temporary file and lost the original ownership and permissions. A working executable could stop being executable; a host-owned file could become difficult for its owner to edit.

The correction lived in the shared file-replacement helper. Preserve the existing metadata before publishing the replacement. Verify it against actual host access.

No amount of “be careful when editing” would have repaired that helper.

## Changing the interface can expose a different problem

Our live integration checks also exposed how native tool calls, commentary, and history interact when you use Codex through a subscription, wired up to a custom harness.

Public commentary should let you follow the work while the agent continues using tools. If it is mistaken for a final response, the loop can end at exactly the wrong moment. If the display drops it, the work becomes harder to follow. If replay drops the accompanying native items, the next turn loses part of the provider's original context.

We corrected those boundaries: streamed public commentary into the generation display, kept reasoning summaries separate, and preserved eligible native items during Responses history replay. The shared history remained API-agnostic.

We tested ordinary Codex chats, parallel Agent Zero workers, and A0 API compatibility. A matched cache check still reached roughly 93% on warm requests. Intermittent zero-cache requests remained unresolved, and we claimed no cache-efficiency gain from replay.

These were verified correctness improvements. We have not established a Terminal-Bench score increase from them.

This is another part of agentic engineering that a single prompt box hides: the difference between what the provider emits, what the agent executes, what the user sees, and what the next turn receives.

## Why Agent Zero is a good place to learn this

If your goal is to learn agentic engineering, we think Agent Zero is one of the best places to start. The reason is how directly you can inspect and change the system.

You can read the prompt that steers a decision. Override it for a profile. Change a plugin for every part of the framework. Inspect an operation and its result. Try a different division of work. Keep the model fixed while testing a hypothesis about its surroundings.

The same framework can be your everyday assistant and your experimental workbench.

That access matters because some intuitively attractive ideas will make things worse. An extra planning step can consume attention without improving execution. A shorter trajectory can mean premature stopping. More recalled material can introduce irrelevant context. A recovered tool call can still leave the user's request unfinished.

An open implementation lets you investigate those differences instead of explaining every outcome as “the model was good” or “the model was bad.”

A literature research informed the experiments. SWE-agent (https://arxiv.org/abs/2405.15793) studies the effect of agent-computer interfaces; Agentless (https://arxiv.org/abs/2407.01489) explores a simpler localization-and-repair workflow; GEPA (https://arxiv.org/abs/2507.19457) develops reflective prompt optimization. They offer useful hypotheses and methods. Their results still need to be tested in your own setup.

## Start with one behavior you can observe

Take a task where your agent repeatedly disappoints you. Save the instruction, starting files, model configuration, and trace.

Locate the failure. Did it misunderstand the requirement? Read the wrong thing? Use an unsupported operation? Lose state? Stop before finishing?

Change one relevant part of the harness. Run it again from the same starting point. Check task success before celebrating fewer tokens. If it improves, repeat the comparison and make sure an existing success still works.

You do not need to build a new framework to begin. You need access to the one you are already using.

Open the harness. Make one change. Find out what it actually did.

Explore Agent Zero · Read the source · Explore Revolve (https://agent-zero.ai/) (https://github.com/agent0ai/agent-zero) (https://github.com/agent0ai/revolve)