← All explainers
A field guide to spoken AI

Realtime voice AI is a conversation carried by events.

Audio streams in while audio streams out. Between them, the system detects turns, updates the conversation, calls tools when needed, and reacts when a person interrupts.

01 · Mental model

A live duet, not exchanged voice memos.

The analogy

Two musicians listen and respond continuously. Timing, pauses, and interruptions are part of the performance.

The reality

A persistent session carries small audio and control events in both directions instead of waiting for one complete recording.

02 · Live loop

Listen, understand, respond—while the clock runs.

INPUTCaptureMicrophone audio becomes a stream.
TURNDetectVAD estimates speech start and stop.
MODELRespondThe conversation produces audio events.
OUTPUTPlayAudio is buffered and rendered.
CONTROLInterruptNew speech can cancel playback.

Realtime is a session, not one request. State evolves as audio and control events arrive.

03 · Try the turn

Watch one conversational timeline.

USER SPEAKS
MODEL
AUDIO RESPONSE
audio.deltaMicrophone frames enter the session.
speech.startedTurn detection marks the beginning of speech.
04 · What travels

Audio plus orchestration.

AUDIOSmall encoded input and output chunks
EVENTSSession updates and response lifecycle
TOOLSStructured function arguments and results
STATEConversation items and instructions
05 · Connection choices

The transport follows the product surface.

WEBRTCBrowser and mobile clients

Designed for realtime media transport and resilient playback.

WEBSOCKETServer-to-server

Direct event stream when your backend controls the audio pipeline.

SIPTelephone networks

Connect voice agents to conventional calling infrastructure.

06 · Failure modes

Every pause is product design.

Network jitter creates gaps. Aggressive turn detection cuts people off. Slow tools stretch silence. Echo can look like a new speaker. Playback must be truncated when the user interrupts so session state matches what they actually heard.

07 · Keep this model

Stream the media; coordinate with events.

HEARInput stream

Capture audio continuously.

DECIDESession state

Detect turns and produce responses.

ADAPTControl events

Handle tools, errors, and interruption.