Build realtime voice agents on AI Gateway

AI Gateway now supports audio/voice. You can add realtime voice, text to speech, and speech to text with the same calls you already use for text, image, and video, routed through AI Gateway alongside every other modality.

Audio launches with models from OpenAI and xAI. Each call gets the same provider routing, observability, spend controls, and bring-your-own-key support you already use for your other models.

These capabilities are in beta and available in AI SDK 7.

Capability	How it works	Use it for
Realtime voice	Live audio in and out, for streaming, low-latency session	Two-way voice agents and live conversation
Text to speech	Text in, audio file out, single request	Voiceovers, spoken responses, audio versions of written content
Speech to text	Recorded audio in, text out, single request	Transcribing voice notes, call recordings

Getting started

Realtime, speech, and transcription model are supported on AI SDK 7.

Realtime voice agents

Realtime turns your app into something a user can hold a conversation with. When they speak, the model responds right away. Because it replies in the moment instead of waiting for a full turn, users can interrupt and talk over it the way they would with a person. It fits voice assistants, customer support agents, hands-free tools, and anywhere a user would rather talk than type.

What sets it apart from chaining models together is that a single realtime model hears audio and produces audio directly, instead of running a speech-to-text, then language model, then text-to-speech pipeline.

In the browser, the useRealtime hook manages the WebSocket connection, microphone capture, and audio playback.

The connection is authenticated with your AI Gateway credential, so you mint a short-lived token on the server and hand the browser only that token. Your API key never reaches the client. Add a route that mints the token:

Then connect from a client component:

The hook captures the microphone, streams the audio to the model through AI Gateway, and plays back the spoken reply. Outside the browser, you can drive the session over a WebSocket yourself with getWebSocketConfig, serializeClientEvent, and parseServerEvent. See the realtime reference for that path.

Inside a realtime session

A realtime session works differently from a normal model call:

• Turn-taking and interruptions. turnDetection: { type: 'server-vad' } lets the server decide when the user has stopped speaking, and lets the user talk over the model to cut a reply short (barge-in), with no client-side silence timers.

• Tools mid-conversation. The model emits a tool call mid-reply, you run it and return the result as a client event, and it folds the answer into what it says next without ending the turn.

Text to speech

Generate spoken audio from text with generateSpeech. Pass a voice and an output format, then write the result to a file:

Speech to text (transcription)

Transcribe recordings into text with transcribe. The audio can be a buffer, a base64 string, or a URL:

Speech and transcription are complementary, so they compose. You can generate audio with one model and read it back with the other, which is a quick way to check both ends of an audio pipeline.

Playground

You can also try audio models without writing any code. Open the models page, click into a model, and interact with it right in the browser. Talk to a realtime model to hold a voice conversation, or send text or audio to a speech or transcription model and read or play back the result.

Routing audio through AI Gateway

Audio calls behave like every other model call on AI Gateway. You use one API key across providers, see requests and usage in observability, apply the same budgets and spend limits, and bring your own provider keys when you need to. Adding speech to an app that already uses AI Gateway for text/images/videos can now all be done in the same place.

More information

• Realtime quickstart

• Speech quickstart (text to speech and speech to text)

• Realtime reference

• See all audio models

Read more