OpenAI Releases GPT-Realtime-2.1 and Mini Models

OpenAI released two new Realtime models, GPT-Realtime-2.1 and GPT-Realtime-2.1-mini, on its API, designed for low-latency voice and multimodal experiences. The GPT-Realtime-2.1-mini model is a compact reasoning engine specifically for real-time voice interactions, offered at the same cost as the previous GPT-Realtime-mini. This release also includes a reduction of at least 25% in p95 latency across all Realtime voice models, achieved through enhanced caching mechanisms.

The GPT-Realtime-2.1-mini model processes both audio and text inputs through a live connection, functioning as a single model for audio processing and generation. This integrated approach eliminates the need for separate speech-to-text and text-to-speech systems, thereby minimizing latency and preserving vocal nuances. A key feature is its reasoning capability, allowing the model to perform internal thought processes before generating a response. The mini tier also supports tool use, enabling the model to plan actions, call external functions, and then provide an answer.

The larger GPT-Realtime-2.1 model is an update to GPT-Realtime-2, featuring improved alphanumeric recognition, better handling of silence and noise, and refined interruption behavior. It supports speech-to-speech communication with adjustable reasoning effort, instruction following, and tool use capabilities. OpenAI suggests using GPT-Realtime-2.1 for applications requiring the most robust real-time reasoning, tool integration, and advanced voice agent functionalities, while GPT-Realtime-2.1-mini is recommended for scenarios prioritizing speed and cost efficiency.

These advancements address a common issue in voice agents where they become unresponsive during tool calls, leading users to believe the connection has dropped and causing interruptions. By incorporating reasoning and a spoken preamble, the models can manage these tool interactions more effectively, preventing partial results and maintaining a coherent conversational state.