Voice AI benchmarks shift to continuous conversation metrics

Late September 2026 marked a visible pivot from turn-taking voice systems toward continuous speech, as AI Engineer argued on September 15 that voice agents succeed only when “all of these have to be well aligned” on a shared timeline rather than treated as discrete chat turns. That framing was quickly reinforced on September 22, when the GPT-Live explainer described “three generations of architecture” culminating in full-duplex design, explicitly positioning continuous audio processing as the next stage beyond turn-based systems that wait for a detector before the main model can respond.

The coordination mattered because the same late-September discussion converged on the practical reason for the shift: turn-based designs were no longer acceptable for natural, low-latency voice interaction. AI Engineer emphasized that voice systems must track events on the timeline of “waveforms” rather than depend on visible text history, while the September 22 GPT-Live explainer argued that turn detectors create awkward delays, force separate interruption handling, and miss or muffle overlapping speech—exactly the failures that continuous, multi-party, full-duplex systems are meant to eliminate.

What makes the new architecture work is not just a better model, but a cleaner division of labor: keep the live path for media and inference only, and push everything variable-latency behind an asynchronous boundary. OpenAI’s July architecture analysis showed why that matters at scale, separating a stateless relay from a stateful transceiver so packet routing stays fast while session logic stays anchored; the result, it said, “handles 900 million weekly users on a relatively small relay footprint,” using “Userspace Go, SO_REUSEPORT, thread pinning, and careful memory” discipline to preserve low-latency continuity.

That separation only pays off if the system is built around continuous delivery, and OpenAI’s August disclosure made the requirement explicit: “a live media system needs to deliver every audio frame on schedule.” The team said, “Over the last six months, we reworked model inference, context management, and media transport to keep speech flowing smoothly from end to end,” a build that demonstrates how continuous media streaming, stateful inference, and asynchronous delegation can be engineered to feel “truly live” while supporting scalable real-time use. ByteByteGo added that “Standard WebRTC costs six round trips… On a mobile network where one round trip takes 60 milliseconds, setup alone can cost over a third of a second,” while OpenAI built “WARP (WebRTC Abridged Roundtrip Protocol)” that “shrinks” setup to one, removing startup drag before the stream even begins. The reason benchmarks matter here is that they now validate whether these architectural choices survive real conversational messiness, not just lab demos. Realtime-Venus shows the link directly: on Full-Duplex-Bench v1.5 it “responds to 75% of user interruptions and achieves continuation rates of 97%, 88%, and 86% under backchannels, other-directed speech, and background speech,” and “exceed[s] Gemini 3.1 Live and GPT-4o on all three continuation metrics”; STEP Audio 3 Realtime similarly frames the target as a continuous loop that coordinates listening, reasoning, and action in real time.

Turn-based benchmarks fail in full-duplex voice because the object being evaluated is no longer a sequence of clean text exchanges but a live, continuous audio interaction. As AI Engineer argued, “there’s no concept of pure kind of text to text here anymore,” so evals must focus on “that entire conversation,” with transcription still useful for auditability and observability but no longer sufficient as the unit of judgment for whether the system behaved well in real time.

What matters instead is whether the system manages the stream correctly moment by moment: staying silent when it should, detecting endpointing, handling barge-in, and recovering when the stream degrades. ByteByteGo noted that “with a full-duplex architecture, there are no turns,” only continuous audio, which is why stream health must be judged with tail-latency and recovery behavior: “p95 is not reliable in full-duplex architectures… a p95 event happens every 20 inferences… Even p99 events show up regularly,” so systems must be designed around p999 and fast recovery.

What makes these benchmarks notable is that they are aimed at deployment failures, not lab-clean turn accuracy. According to Daily Paper Cast, EVA-Bench “orchestrates fully automated bot to bot audio simulations over dynamic multi turn dialogs,” explicitly generating full conversations so evaluators can see whether an agent maintains context, recovers from misunderstandings, and actually resolves tasks under realistic variation, including accents and background noise; it also measures more than completion, because a system can finish a task yet still mishandle turn-taking, violate policy, or overwhelm the user.

Duplex-MPE pushes even closer to the hard edge of real use by stripping away the scaffolding models often rely on. Hugging Face Daily Papers says, “The benchmark contains 2,000 scenarios with three or four human speakers and one assistant, each paired across explicit and implicit addressing of the same request,” while Daily Paper Cast adds that this yields “4,000 audio streams” and that metrics include “fresh onset response rate,” “conditional answer accuracy,” “silence preservation,” and “answering window yield,” testing whether the assistant starts within five seconds, stays quiet when appropriate, and stops after resolution.