How Vercel replaced multi-agent prompt chains with a sandboxed file-system execution loop to double benchmark performance.
Adapted from @monokern# The File-System Agent Pattern: Why Multi-Agent Pipelines Fail and How Vercel Built Eve How moving from multi-agent prompt chains to sandbox file systems and distilled skills doubles eval benchmarks and scales to production. # The Architectural Trap: Over-Engineered Multi-Agent Workflows vs. Blind Mega-Prompts Most internal agent initiatives fall into one of two failure modes. Either teams paste a massive database schema into a single mega-prompt, relying on human copy-pasting to execute generated SQL, or they over-engineer fragile multi-agent pipelines where specialized agents pass lossy textual summaries down a strict assembly line. The first approach hits an immediate context ceiling, generating syntactically invalid queries as soon as schema complexity grows. The second approach creates state isolation: when an execution sub-agent hits a database error, it cannot reflect on the initial planning steps because its context window only holds a distilled, lossy summary from the previous node. When Vercel built their internal data science agent (D0) to serve growing internal analytics demands across marketing, sales, finance, and legal teams, they tracked this exact evolution. The initial single mega-prompt model failed on complex joins. The multi-agent pipeline—splitting responsibilities across dedicated Query, Planning, Execution, and Reporting agents—improved pipeline execution but capped evaluation performance around 30%. The breakthrough didn't come from adding more specialized agents or expanding system prompts. It came from fundamentally refactoring the agent runtime into an isolated file-system sandbox running a single, self-correcting execution loop. # The Economics of Execution: From 400 Lines of Provider Code to Generic Sandboxes To understand why traditional agent abstractions fail in production, you have to look at the honest math of context management and API coupling. Legacy agent implementations force engineering teams to maintain 300 to 400 lines of provider-specific boilerplate just to handle execution loops, model switching, and tool calling across different LLM vendors. When provider APIs change, those 400 lines break. More critically, hardcoding bespoke, prescriptive tool definitions for every possible sub-task degrades model performance. When an agent is given dozens of hyper-specific tool definitions, tool selection accuracy drops dramatically, and context windows fill with schema overhead rather than execution history. The alternative is unifying the execution layer around two primitives: unified model abstractions (swapping provider-specific plumbing for single-line model interfaces) and sandbox execution environments. Instead of writing custom API wrappers for every database look-up or file parser, you supply the agent with a sandboxed file system and a generic terminal execution tool. By dumping the entire semantic data layer (such as entity YAMLs and join definitions) directly into a sandboxed file system, the agent uses standardized file-system primitives (grep, cat, ls, and bash) to explore schemas dynamically. In Vercel's real-world benchmarking, switching from rigid tool chains to a sandboxed file-system agent inspired by Claude Code and Opus 4.5 instantly doubled eval benchmark pass rates. # The File-System Agent Pattern: The 5-Phase Evolution The path to building production-ready business agents follows five distinct architectural phases. Skipping phases or starting with complex multi-agent frameworks almost guarantees fragile, unmaintainable code. ## Phase 1: The Monolithic System Prompt The team passes a raw Snowflake database schema and user query into a single LLM prompt. The model outputs raw SQL, which a human engineer manually reviews and executes. This validates whether the model possesses sufficient domain intelligence, but fails to automate end-to-end workflows. ## Phase 2: Chained Multi-Agent Pipelines Work is decomposed into isolated node agents (Query -> Planning -> Execution -> Reporting). Each agent owns a tightly scoped system prompt and dedicated tools (e.g., read_entity_yaml, search_schemas). While end-to-end execution is achieved, state isolation prevents recovery when errors occur during execution. ## Phase 3: The Single State-Managing Loop The architecture consolidates into a single, high-capacity agent with a max execution budget (e.g., maxSteps: 100). The agent manages its own internal state across planning, building, executing, and reporting phases. Because full execution history remains in context, the agent can reflect on SQL join errors and retry without human intervention. However, handling edge-case user queries remains constrained by static context, keeping evals capped around 30%. ## Phase 4: The Sandboxed File-System Agent The single agent is connected directly to a sandboxed execution environment containing the project's entire semantic metadata layer. Equipped with a minimal bash_tool, the agent navigates directories, inspects schema files, writes execution scripts, and validates SQL queries autonomously. This shift doubles evaluation pass rates by leveraging pre-trained terminal and file manipulation capabilities. ## Phase 5: Pattern Distillation via Modular Skills As query volume scales to thousands of runs per day, recurring query patterns (e.g., customer churn metrics, billing lookups, NPM download aggregation) are automatically distilled into structured markdown skill files saved in a /skills directory. The agent reads these skills at runtime, avoiding cold-start context discovery penalties. # System Blueprint: Framework-Defined Infrastructure with Eve To make file-system agent patterns reproducible, Vercel built Eve—a framework that acts as the "Next.js for Agents." Just as Next.js introduced file-system routing for web applications (routing pages to CDNs and API routes to serverless functions), Eve enforces convention-over-configuration for agentic infrastructure. An Eve agent organizes codebase infrastructure into declarative directory conventions: ## Protocol Framework: The Core Runtime Primitives 1. Isolated Execution Runtime: Agent steps execute inside isolated micro-sandboxes (like Vercel Sandbox or Docker) to ensure secure file-system isolation and prevent arbitrary code execution leaks. 1. State Durability: Complex multi-step reasoning runs require persistent step state. Eve leverages durable execution primitives (e.g., Vercel Workflows) to pause, resume, and retry steps across network or model timeouts without dropping context. 1. Zero-Trust Connections: Database access and sensitive API calls utilize short-lived OIDC tokens generated dynamically via identity services (e.g., Vercel Connect), eliminating long-lived credentials inside context prompts. 1. Skills Injection Engine: Domain-specific procedures live in /skills/*.md. When a user query matches a known pattern, the runtime injects the relevant skill file, reducing required reasoning steps and preventing model token blowout. # Tactical Implementation: From Unstructured Queries to Distilled Skills When building internal agents for domain-specific tasks—whether for legal contract redlining, marketing retro analysis, or Snowflake queries—you must avoid treating every query as a blank slate. ## The Cold-Start Trap vs. The Skills Pattern In an unoptimized setup, an agent receiving a request like "Calculate Q3 enterprise renewal drop-offs" spends 80% of its execution steps discovering table relationships, finding foreign keys, and guessing business metrics. By introducing a /skills directory, high-frequency analytical patterns are codified into markdown SOPs that the agent reads before generating execution plans. By maintaining roughly 100 domain skills in production, Vercel reduced query failure rates and minimized execution token costs across thousands of daily employee queries across sales, finance, legal, and engineering teams # Failure Modes & Diagnostic Guide When deploying file-system agents into production, teams routinely encounter four critical diagnostic failure modes: ## 1. The Context Leak Failure - Symptom: The model ignores critical guardrails, leaks internal API keys, or executes unauthorized administrative operations. - Root Cause: Over-allocating universal system prompt space instead of isolating sensitive tools behind authorization checks, or leaving long-lived credentials hardcoded in context. - Fix: YOU MUST route external connections through short-lived OIDC access tokens (Vercel Connect) and restrict file-system sandboxes to read-only mounts for production environments. ## 2. The Multi-Agent Summary Degradation - Symptom: Downstream agents output completely hallucinatory recommendations despite receiving valid data from upstream steps. - Root Cause: Splitting tasks across rigid, sequential sub-agent chains where intermediate nodes output brief string summaries, stripping structural context and error stack traces. - Fix: Replace multi-agent pipelines with a single stateful execution loop operating over a shared local sandbox file system (maxSteps: 100). ## 3. The Tool Bloat Paralysis - Symptom: The agent selects invalid tools, enters infinite loops calling the wrong endpoints, or fails tool invocation syntax checks. - Root Cause: Registering dozens of hyper-specific custom tools instead of generic file system and terminal tools. - Fix: Prune tool lists down to core file manipulation primitives (bash_tool, read_file, write_file). Move specialized business logic into executable scripts inside the sandbox or into declarative skill files in /skills. ## 4. The Unpruned Incident-Hotfix Accumulation - Symptom: Token costs explode and system prompt instructions begin contradicting each other, leading to unpredictable agent routing. - Root Cause: Appending system instructions defensively every time an edge-case query fails, creating a bloated system prompt. - Fix: Conduct periodic prompt audits. Delete instructions that the model follows naturally, move task-specific logic into /skills, and run prompts through automated evaluation pipelines. # Setting This Up This Week: The Implementation Roadmap To transition your team from fragile custom scripts to production-grade agents without wasting engineering cycles, follow this phased rollout plan. ## Step 1: Establish the Baseline Evaluation Benchmark (Day 1) - Collect 30 to 50 real-world queries from your target internal users (e.g., sales, finance, data team). - Run these queries through your current baseline prompt or multi-agent setup. - Record pass/fail rates and document exact query failure modes. ## Step 2: Deploy a Sandboxed File-System Runtime (Days 2-3) - Initialize an Eve agent repository structure (eve init or clone a sandbox template from eve.dev). - Populate the sandbox directory with your domain's semantic metadata layer (YAML definitions, schema docs, markdown SOPs). - Attach the standard bash_tool primitive to the agent, granting read/write access strictly inside the micro-sandbox. - Re-run your 50-query evaluation set. Target an immediate doubling of baseline pass rates. ## Step 3: Implement The Recurrent Skills Pipeline (Days 4-5) - Analyze the successful execution logs from Step 2. - Identify top query clusters (e.g., aggregation queries, lookup functions, reporting tasks). - Draft standardized SOP markdown files for the top 5 query types and place them into /skills. - Connect observability telemetry (tracking step counts, tool calls, and token costs per agent run). ## Step 4: Production Rollout and Self-Service Scale (Week 2+) - Expose the agent to internal user channels (Slack, CLI, or Web UI). - Implement automated background distillation jobs to turn high-frequency successful runs into new /skills files. - Maintain a target of zero long-lived credentials by enforcing short-lived OIDC authentication across all internal database integrations.
