Voice Agent Latency Benchmarked Beyond Time To First Token

Time to First Token (TTFT) is a commonly used metric for evaluating inference APIs in voice applications, but it is insufficient for accurately measuring user experience, according to a new benchmark analysis. TTFT measures the time elapsed between sending an inference request and receiving the first token back, marking the moment a system becomes visibly active. However, for voice agents, this metric does not capture the full user-perceived latency because text-to-speech (TTS) models require complete clauses or sentences to generate audio. This means a voice agent cannot begin speaking until a full sentence is synthesized, even if the initial token is received quickly.

LiveKit, in its analysis, introduces Time to First Sentence (TTFS) as a more relevant metric, arguing that it better reflects what users actually experience. TTFS accounts for both the speed at which generation begins (controlled by TTFT) and the speed at which the first complete sentence is delivered (controlled by tokens per second). A voice agent provider must excel in both aspects to feel genuinely fast to the user. The benchmark highlights that optimizing only TTFT can lead to a misleading perception of performance if the TTS component is slow to produce a coherent output.

The latency budget for a voice turn involves multiple stages, each contributing to the overall delay. LiveKit's breakdown estimates Speech-to-Text (STT) at approximately 100–200 milliseconds, the Large Language Model (LLM) with streaming at 300–500 milliseconds, TTS at 100–200 milliseconds, and network latency over WebRTC at 50–150 milliseconds. This results in a practical end-to-end target latency of 700 milliseconds to 1.2 seconds for a seamless conversational experience. Kwindla Hultman Kramer, co-creator of Pipecat, has previously advised targeting a median voice-to-voice latency of 800 milliseconds, with an acceptable upper limit of 1,500 milliseconds for proof-of-concept stages. Kramer's rough calculation divides this latency budget into four equal parts of approximately 200 milliseconds each, covering transport and media processing, STT, and phrase end detection.

The benchmark emphasizes that a voice agent's responsiveness is a sum of its parts, and focusing solely on TTFT overlooks critical components like TTS synthesis time. For an agent to feel truly conversational, the entire pipeline, from speech input to text processing and finally audio output, must operate within a tight latency budget. The analysis aims to provide a more granular understanding of where milliseconds are spent in the voice agent stack, enabling developers to make informed decisions about API selection and system design to minimize perceived delays and enhance user interaction quality.