Voice API

Voice API uses two steps: your backend creates a session with a Developer API key, then the client connects over WebSocket with a short-lived token. Never send the Developer API key to a browser.

1. Allow the browser originCopy Anchor Link

For a browser client, go to Developer API → Voice Browser Access and add the website's exact origin. The origin must use http or https; its scheme, hostname, and effective port must all match.

This list is independent from the websites allowed to load the Web Widget. Server, mobile, and desktop clients that omit Origin do not require this step, but they must still present a valid session token.

2. Create a session from your backendCopy Anchor Link

POST /v1/voice/sessions · scope voice:connect

curl "$ASSISTANT_API_URL/v1/voice/sessions" \
  --request POST \
  --header "Authorization: Bearer $ASSISTANT_API_KEY" \
  --header "Content-Type: application/json" \
  --data "{
    \"external_user_id\": \"$EXTERNAL_USER_ID\",
    \"mode\": \"realtime\",
    \"turn_detection\": \"server_vad\",
    \"max_session_seconds\": 60,
    \"input_audio_format\": {
      \"codec\": \"pcm_s16le\",
      \"sample_rate_hz\": 16000,
      \"channels\": 1
    },
    \"output_audio_format\": {
      \"codec\": \"opus\",
      \"sample_rate_hz\": 24000,
      \"channels\": 1
    }
  }"

201 response:

{
  "id": "sess_01K...",
  "token": "short_lived_single_use_token",
  "expires_at": "2026-08-13T08:32:00Z",
  "websocket_url": "wss://api.assistantcore.com/v1/voice"
}

Return only id, token, expires_at, and websocket_url to the authenticated client. Never return the Developer API key. The token expires after about 120 seconds and can be used once.

Mobile and desktop applicationsCopy Anchor Link

Do not embed a Developer API key in an APK, IPA, or desktop binary. A native application is a public client, and a secret shipped with the application can be extracted.

Your backend keeps the Developer API key, authenticates the app user, creates a voice session, and returns only token, expires_at, and websocket_url to the application. Store a per-installation refresh token or private key in Keychain/Keystore when your application has its own sign-in flow; do not use one shared Developer API key as the app credential.

3. Open the WebSocketCopy Anchor Link

Connect to the exact returned websocket_url. The first event, sent within 15 seconds, must be:

{
  "type": "session.start",
  "token": "short_lived_single_use_token"
}

Do not put the token in the URL. The server sends session.created when ready:

{
  "type": "session.created",
  "sequence": 1,
  "session_id": "sess_01K..."
}

Every JSON server event has a monotonically increasing sequence. Binary server messages contain audio in the negotiated output format.

The browser connection skeleton below handles session and control events only; capture and playback depend on your application's audio stack:

const session = await fetch("/api/assistant/voice-session", {
  method: "POST",
}).then((response) => response.json())

const socket = new WebSocket(session.websocket_url)
socket.binaryType = "arraybuffer"

socket.addEventListener("open", () => {
  socket.send(JSON.stringify({
    type: "session.start",
    token: session.token,
  }))
})

socket.addEventListener("message", (message) => {
  if (typeof message.data === "string") {
    const event = JSON.parse(message.data)
    console.log(event.type, event.sequence)
    return
  }

  // Queue message.data for playback using the negotiated output format.
})

socket.addEventListener("close", () => {
  // Discard this token. Ask the backend for a new session before reconnecting.
})

Send textCopy Anchor Link

{ "type": "input_text.create", "text": "What is your return policy?" }

Do not send new input while a response is active. Send response.cancel first for barge-in.

Send audioCopy Anchor Link

For standard + manual or realtime + manual, use this order:

1. Send {"type":"input_audio.start"}.
2. Send one or more binary WebSocket messages in the negotiated input format.
3. Send {"type":"input_audio.commit"}.

For standard + auto or realtime + server_vad, send input_audio.start once after session.created, stream binary audio continuously, and do not send input_audio.commit. The server or provider emits speech boundaries and creates exactly one response per detected turn.

For pcm_s16le, the binary payload is raw signed 16-bit little-endian mono PCM without a WAV header. Prefer short, regular frames; each binary message must be at most 256 KiB. For Opus, each binary message contains an Opus frame at 24 kHz mono.

Client eventsCopy Anchor Link

Server eventsCopy Anchor Link

Latency metricsCopy Anchor Link

With include_metrics: true, each completed response is followed by one response.metrics event. Cancelled or failed responses do not emit it.

{
  "type": "response.metrics",
  "sequence": 42,
  "message_id": "7c1e...",
  "end_of_utterance_ms": 610,
  "transcription_ms": 180,
  "llm_ttft_ms": 420,
  "tts_ttfb_ms": 150,
  "e2e_latency_ms": 1390
}

All values are integer milliseconds measured on the server, so they exclude network time to your client. A field is omitted when the stage was not measured. Realtime mode runs recognition, reasoning, and speech inside the provider, so it reports only e2e_latency_ms.

Cancel and barge-inCopy Anchor Link

{ "type": "response.cancel" }

After canceling, stop playback and discard buffered output audio, then wait for response.completed with status cancelled. Send new input only afterward.

ReconnectCopy Anchor Link

A session token cannot be reused. After a lost connection:

1. Close the client socket and discard the old token.
2. Ask your backend to create a new session.
3. Include the previous conversation_id in the session request to resume it.
4. Connect to the new URL and send the new token.

Do not retry forever. Use exponential backoff for retryable failures and stop when the credential, origin, plan, or credits are invalid.

Current connection limitsCopy Anchor Link

• JSON event: 16 KiB maximum; binary frame: 256 KiB maximum.
• Up to 50 events/second and 10 MiB/minute per connection.
• Up to 15 minutes of input audio, 30 minutes/session, and 2 minutes idle. A smaller max_session_seconds is enforced when issued in the session token.
• Realtime sessions are also capped at what the owner's credits pay for: the issued max_session_seconds never exceeds that budget, and a session ended by it closes with INSUFFICIENT_CREDITS rather than SESSION_DURATION_EXCEEDED.
• Up to 5 connections/credential and 50 connections/assistant.

Limits may change during Preview. See Errors and limits for close-code handling.