OpenAI has released its Realtime API, which connects AI models to apps for live interactions, for general use. The company has also released a new speaking model called gpt-realtime.
The Realtime API and gpt-realtime model are designed to power AI voice agents for use cases such as customer support, where a customer calls a helpline and speaks directly with an AI agent. While other APIs combine separate speech-to-text and text-to-speech models, the Realtime API operates as a speech-to-speech system, reducing latency and capturing subtle conversational cues for more natural interactions. With its broader release, the Realtime API introduces new features, including image inputs, reusable prompts, and phone connectivity.
The gpt-realtime model also delivers significant improvements over its predecessor, gpt-realtime-preview, in instruction following, tool use, and generating speech that sounds more human-like.
“Natural-sounding conversation is critical for deploying voice agents in the real world,” OpenAI said in a blog post. “Models need to speak with the intonation, emotion, and pace of a human to create an enjoyable experience and encourage continuous conversation with users.”
Realtime API’s new features include allowing users call an AI with their phone
Four new features have been added to the Realtime API: support for Model Context Protocol (MCP) servers, image inputs, phone calling through Session Initiation Protocol (SIP), and reusable prompts.
The API allows a session to connect to a remote MCP server, a plug-in that provides extra tools or skills, by passing its URL into the configuration. Once connected, any tools from that server become available right away without manual integration.
Images, photos, and screenshots can now be sent to a Realtime API session alongside audio or text. This enables the model to understand and respond based on what is visually shown, such as reading text in a screenshot or describing the contents of a picture.
The Realtime API’s SIP support lets it connect directly with phone networks, PBX systems, and desk phones, allowing users to have a real-time phone conversation with the AI. It also lets users save and reuse prompts across sessions.
gpt-realtime can sense laughter, change its tone from direct to warm, and much more
OpenAI says that gpt-realtime is its most advanced speech-to-speech model yet, with improvements in audio quality, intelligence, instruction following, and function calling.
The model has been trained to produce higher-quality speech and can respond to prompts asking it to alter its sound, such as “speak quickly and professionally” or “speak empathetically in a French accent.” Accuracy on the MultiChallenge benchmark, which measures performance in multi-turn conversations, has improved by 10.1% compared to the December 2024 release.
gpt-realtime also responds more intelligently to voice commands, picking up non-verbal cues like laughter, switching languages mid-sentence, adapting its tone to the context of the conversation, and detecting alphanumeric sequences in different languages. The update has boosted its score on the Big Bench Audio benchmark, which assesses the reasoning capabilities of language models that support audio input, by 17.2%.
Stronger adherence to instructions and improved function calling
OpenAI has improved gpt-realtime’s adherence to its master prompts so that even subtle instructions are more reliably followed throughout a conversation. This ability is measured by the MultiChallenge audio benchmark, which rose by 9.9%.
Finally, gpt-realtime’s function calling has been enhanced so that the tools it selects are more relevant, invoked at more appropriate times, and supplied with more accurate data. It is also better at asynchronous function calling, meaning that it can continue a fluid conversation while waiting on the results of a function call.
All eight of the model’s existing voices will be equipped with these updates, as well as two new ones named Cedar and Marin.
Pricing details and availability
Both the generally available Realtime API and the gpt-realtime model are now live. Pricing for gpt-realtime is structured as follows:
- $32 per million audio input tokens
- $0.40 per million cached input tokens
- $64 per million audio output tokens
Developers can also set token limits and truncate multiple turns at once to reduce the cost of longer sessions.
Read a deep dive on these OpenAI releases on our sister site The Neuron.