Sesame TurnBench Exposes Gemini and OpenAI Timing Flaws

Sesame has released TurnBench, an open-source benchmark designed to evaluate how effectively voice AI models manage the natural flow of conversation. The benchmark includes a 30-hour evaluation corpus of dual-channel dyadic English speech, consisting of 154 dialogues featuring 106 voice actors in 53 pairs. Triple-annotated with a strong Fleiss' kappa of 0.78, the dataset is balanced across six conversational registers. Alongside the evaluation set, Sesame released otoSpeech, a 104-hour hand-labeled training set on Hugging Face, and made the scorer available on GitHub.

The benchmark's initial leaderboard reveals that even the most advanced commercial and open-source models struggle with conversational rhythm. Voice Activity Projection led the pack with an End-of-Turn recall of 0.845, a false-positive rate of 0.055, and a median latency of 368 milliseconds, though it still takes nearly a second to commit interruptions. In comparison, ESPnet Turntaking achieved a 0.826 End-of-Turn recall with 862 milliseconds of latency, and Kyutai Semantic VAD reached 0.773 recall with 1007 milliseconds of latency. Commercial giants lagged further behind: Gemini 3.1 Live scored 0.657 End-of-Turn recall with 1234 milliseconds of latency, OpenAI Realtime scored 0.303 recall with 793 milliseconds of latency, and Moshi scored 0.233 recall with 702 milliseconds of latency.

For developers building voice products, TurnBench offers a much-needed alternative to outdated datasets like Switchboard. It provides a realistic, held-out test set to evaluate endpointers against commercial systems under a scoring protocol that measures actual committed decisions. Practitioners can also use the 104-hour otoSpeech corpus to train supervised turn-taking models without having to design their own annotation schemes. Ultimately, the benchmark highlights a massive gap between AI and humans, who typically begin turn transfers 151 milliseconds before a speaker finishes.