Explains how OpenAI's new speech-to-speech model is designed for cooperative multi-model systems, enabling non-blocking voice agents that delegate work to backend models.
Adapted from @kwindlaReally cool to see the new GPT-Live-1 model and API in production, today. For me, the most exciting thing about the new model is that it's trained to be used together with other models. This is important because voice agents need to be continuously responsive. You can't block the voice conversation loop. But agents often need to make tool calls, interact with backend systems, and leverage test-time-compute. So GPT-Live-1 natively supports working cooperatively with a backend model. You prompt the frontend model and the backend model separately. Here's part of an example from the GPT-Live-1 docs: ``` Delegation policy: Backend tools: - Appointments: check available times and create, change, or cancel bookings. Delegate to the backend when: - The user asks for availability or wants to create, change, or cancel a booking. - A correction changes a booking task already in progress. - The answer needs careful reasoning beyond a simple reply. ``` Multi-model systems are the future of software. Almost everything I build for myself, now, and most of the work I do with customers and partners, involves multiple models, multiple inference loops, multiple prompts, and multiple layers of context management/sharing. That's true whether I'm building voice agents, desktop software, or duct taping together management tooling for all the coding agents I have running all the time across various machines in my house and in the cloud. We're just starting to explore the abstractions that make it easy and productive to distribute work between multiple LLMs and coordinate their output. OpenAI's "delegation" pattern is one such abstraction. It's great to see this pattern trained into a production model, so that we can leverage it natively at the prompt level. GPT-Live-1 is also the first production speech-to-speech model that can talk and listen at the same time. That ability unlocks better turn detection and more natural voice behaviors (like making small filler sounds that let people know you're listening to them, called "backchanneling"). Several research models have implemented full-duplex audio streaming, most notably the Moshi model from Kyutai. If you are interested in LLM architectures and you haven't spent much time with the 2024 paper about Moshi, stop reading this and go load that paper up in your web browser. I promise it's worth it. But Moshi was a much smaller model than GPT-Live-1, and we've had two years of AI research progress in general since Moshi was released. Full duplex is a hard problem, and it's wonderful to see a model that pushes the frontiers of audio capabilities, released as a general availability product, served by a production API. And speaking of the API, the OpenAI team has evolved the realtime API to support a complete set of context engineering and session management capabilities. We can now build things like state machine conversation graphs, threaded/backtracking modes, and structured data input that were difficult with the first generation of speech-to-speech APIs. Congratulations to everyone at OpenAI who worked on this model and API. It's great to see. Er, to hear. :-)
