Code-as-Policy for Production Robotics

# Code-as-Policy: Sisyphus Deployed

There has been a lot of buzz recently around code-as-policy and, more broadly, the idea of robot-use agents: general-purpose models learning to operate robots in much the same way they already operate browsers, terminals and computers [1].

The progress here has been incredible. GPT-6 Astra was able to control YAM robot arms from camera views and end-effector commands, succeeding 19 out of 20 times on a simple block-into-bowl task, although only 2 out of 20 times on a much more precise puzzle-piece insertion task [2]. Anthropic has explored the same broader idea across several robots and control interfaces [3].

But there is a big difference between a robot doing something impressive most of the time and a robot doing economically useful work every day. Production robotics lives in the March of Nines: getting from 90% reliability to 99%, then 99.9%, and continuing to squeeze out the rare failures that only show up after hundreds or thousands of runs.

A behavior that works 9 times out of 10 can make a great demo and still be useless if the tenth failure requires human intervention. This matters especially in science labs (where we deploy). Pouring a tube or moving an object is not the same thing as running an experiment. Real experiments are long chains of dependent steps, and reliability has to hold across all of them, not just the headline manipulation. Once you have a behavior that works, there is enormous value in making it repeatable.

Getting there means building a harness around the model: better tooling, explicit representations of the physical world, ways to verify behavior, and deterministic execution once a workflow has been developed. The agent can still step back in when something unexpected happens, but it does not need to sit in the loop for every successful run.

We have been betting on a more programmatic version of this idea for a while: using agents to write the policies and workflows that operate the robot.

Our bias is toward getting robots into real environments doing useful, high-leverage work as quickly as possible. That pushes us toward systems that can be tested, trusted and repeated in production.

When we did YC about a year ago, we started with fairly basic pick-and-place demos. Since then, we have pushed the same idea significantly further. We now have a toolkit that lets us deploy robots inside scientific labs, build fairly complex workflows, and host hackathons where multiple people can independently use agents to program robots within a few hours.

Along the way, we have learned a lot about how to actually make LLMs and coding agents useful for robotics. The biggest lesson is that code-as-policy works extremely well - but only if you build the right system around it.

## What is Code-as-Policy?

There are roughly three ways foundation models are being used in robotics today.

VLAs map vision and language directly to robot actions through a learned policy. A model such as π0.5 follows this approach [4].

Online LLM control puts a general-purpose model directly in the loop: it observes the scene and repeatedly decides the next robot action.

Code-as-policy moves the model up a level. The model writes executable code that calls perception, planning and control APIs, and that program drives the robot [5].

For us, the production advantage is that code gives us something explicit and modular to inspect, test, patch and make deterministic. With a learned policy, failures can be harder to isolate, and fixing one behavior can affect another. Learned policies and online control can still sit inside the system where useful, while code provides the broader orchestration layer.

## When does Code-as-Policy make sense?

Our goal as a company is to automate science.

That turns out to be a particularly good fit for code-as-policy: the work repeats, and the environment can be represented well enough for the agent to test and verify what it builds. For the kinds of problems we work on, code-as-policy can comfortably handle around 95% of the tasks we encounter.

By deliberately focusing on this domain, we avoid what we call the Grandmother's Cup Problem: the requirement for a robot to walk into your grandmother's house, open an arbitrary cabinet it has never seen before, identify an arbitrary cup, understand how that cabinet and cup behave, and successfully manipulate them. A general-purpose robot has to solve that open-ended case. By choosing a domain, we get to exploit the structure that already exists.

Scientific labs are much more constrained. The set of objects is smaller. Equipment tends to live in known places. Workflows repeat. A plate is still a plate tomorrow. A centrifuge still has the same door, rotor and controls. A pipette still behaves according to the same rules.

That structure gives us a lot of leverage. Instead of asking a model to reason from scratch about the entire physical world, we can give it a known environment and a set of tools for interacting with it.

## Directly plugging an LLM into a robot is a bad idea in production

By “directly”, we mean putting a general-purpose model in the online control loop: raw sensor state comes in, and the model repeatedly emits the next robot action.

These models can already look at a scene, reason about what needs to happen, and produce useful robotic behavior with surprisingly little structure around them. For demos and simple pick-and-place tasks, that can work extremely well.

But production robotics has a much higher bar. For most commercial use cases, a robotic system that is not reliable is not really a tool. It is a toy.

Models still have weak spots in spatial reasoning and physical interaction. They can struggle to reason about how mechanisms articulate, how motions constrain one another, and how an individually sensible movement can leave the robot in a bad configuration.

A language model might understand perfectly well that a centrifuge lid needs to be opened. That does not mean it intuitively understands the geometry of the hinge, the swept volume of the lid, where the robot's elbow will end up while opening it, or whether the gripper can maintain contact throughout the motion.

Those details matter enormously in robotics.

This distinction also matters in the current robot-use-agent discussion. Reasoning about what a robot should do is becoming very strong, while low-level control, dynamics, contact and dexterity remain different problems. Anthropic similarly found that the same models performed substantially better with Python controllers and higher-level robot capabilities than when asked to solve control directly [3].

That has shaped how we think about these systems. We have found it much more useful to treat the LLM as a programmer operating inside a robotics system, rather than as the robotics system itself.

An LLM in the online loop puts its stochasticity directly into your reliability budget. Treating it as a programmer moves that stochasticity into authoring, where it is cheap: write the routine, test it in simulation, then run the verified behavior deterministically.

## What do you need to make Code-as-Policy work?

The way we have approached it looks roughly like this.

1. Build a simulation world for the LLM

We give the model a representation of the environment where it can reason about and test the workflow it is creating. In our system, this is a saved 3D world with a structured text representation containing objects, their poses, collision information and articulated state. This gives the agent somewhere to experiment without immediately turning every mistake into a physical failure (see zeon docs). (https://readme.zeonsystems.app/docs/building-a-world)

2. Make the simulated world match reality

The usefulness of the simulation depends heavily on how well it corresponds to the actual lab. We have spent a lot of time building the perception and localization systems needed to keep the simulated world aligned with reality. Physical objects are localized in the lab and matched to their simulated counterparts, so the same project and workflow can move between a cloud simulation and the physical robot (see zeon docs). (https://readme.zeonsystems.app/docs/key-concepts)

3. Make objects explicitly annotatable

Each object is represented in text and can expose useful poses and interaction points: where the robot should grasp it, where something should be inserted, or which direction the robot should approach it from.

This is how we contextualize the agent with the physical knowledge it needs about that object. Instead of rediscovering the same geometry every time, the model can work from interaction geometry we have explicitly defined (see zeon docs).  (https://readme.zeonsystems.app/docs/skill-authoring-patterns)

4. Give the LLM robotic primitives

We expose reusable robotic primitives that let the model manipulate objects, move safely and interact with the environment. These are ordinary Python functions that become the vocabulary the model uses to build larger behaviors. Low-level controllers that need to run faster or more consistently can be exposed as primitives the agent calls (see zeon docs). (https://readme.zeonsystems.app/docs/authoring-a-skill)

5. Let those primitives be composed as Python

This is where code-as-policy becomes particularly useful. The agent can generate ordinary Python containing loops, functions, conditions, calculations and calls into the robotics toolkit. You get the expressiveness of a general-purpose programming language without asking the model to generate low-level robot commands directly. The resulting program is also inspectable: you can read it, modify it, version it, test it and rerun it (see zeon docs). (https://readme.zeonsystems.app/docs/authoring-a-skill)

6. Give the agent tools to verify and simulate its own work

Generating the program is only half of the problem. The agent also needs tools that help it understand whether what it generated actually works. We give it ways to inspect execution, simplify calculations, test assumptions and navigate the realities of physical execution. The same underlying project can be exercised in simulation and then run against the physical hardware (see zeon docs). (https://readme.zeonsystems.app/docs/running-a-workflow-on-real-hardware)

This creates a development loop much closer to normal software development: build with an agent, inspect it in cloud simulation, iterate on the behavior either manually or with the agent, then run the verified program on the physical robot.

## Does robot learning play any role?

Yes - but we do not think everything needs to be learned.

Code, geometry and deterministic robotic primitives work extremely well for structured tasks, but there are places where that representation starts to break down.

Deformable objects are a good example. Interacting with paper, films or other flexible materials can be extremely difficult to model explicitly. Small differences in contact, friction or deformation can completely change what happens.

For actions like these, we can perform an individual node of the workflow using a fine-tuned model while keeping the broader workflow deterministic. The learned policy simply becomes another primitive that the program can call when explicit modeling becomes painful.

Our current bet is a hybrid: deterministic where we can, learned where we need to. Code-as-policy gives us a useful orchestration layer because a learned policy can sit inside the same workflow as any other robotic primitive.

## Where we have landed

Our view of code-as-policy has changed quite a bit from when we first started.

Originally, the exciting part was that an LLM could write some code and make a robot move. That is still cool, but it now feels like the least interesting part.

What is more interesting is that agents are becoming a genuinely useful way to program robots. If you give them a good representation of the world, useful primitives, simulation, and enough tooling around execution, they can build surprisingly complicated robotic behavior.

There are large parts of physical interaction that these models still struggle with, but the rate of progress is pretty hard to ignore.

A year ago we were excited that an LLM could make an arm pick something up. Today, scientists located across the country independently design and run real, verified experiments on our system.

Now it feels increasingly plausible that programming a robot could become as normal as programming a computer.

References:

[1] https://web.mit.edu/phillipi/www/writing/robot-use-agents.html

[2] https://openai.robocurve.org/gpt-6-astra/

[3] https://www.anthropic.com/research/claude-plays-robotics (https://www.anthropic.com/research/claude-plays-robotics?utm_source=chatgpt.com)

[4] https://www.physicalintelligence.company/download/pi05.pdf

[5] https://code-as-policies.github.io/