Audio

Negotiated formats

The server chooses both directions in conversation_initiation_metadata. The controller validates the values, stores them on inputAudioFormat and outputAudioFormat, and passes them to the audio manager. Never assume 16 kHz.

If the server negotiates a format the browser cannot handle, the start fails with a protocol-error before audio flows. You can validate a value yourself with parseAudioFormat(id).

Capture path

The default path uses an AudioWorklet and falls back to ScriptProcessor when a worklet cannot load. Once audio is flowing, each chunk goes through the same chain:

1. Capture roughly 100 milliseconds of microphone audio. The window is controlled by inputChunkDurationMs.

2. Apply a low-pass filter to reduce aliasing before the sample rate change.

3. Resample from the browser context rate to the negotiated input rate with linear interpolation.

4. Clamp the samples, encode signed 16-bit little-endian PCM, and base64 the result.

5. Send the frame as user_audio_chunk over the existing WebSocket.

Set inputChunkDurationMs between 50 and 250. Values outside that range are clamped. Smaller chunks reduce latency and increase frame rate. The default of 100 is a practical balance for speech.

Microphone capture starts muted when micMuted is true. Muting stops frames without stopping the track, so the browser permission indicator may stay active for the whole session.

Playback path

Agent audio arrives as base64 chunks. The audio manager decodes PCM16 or µ-law, schedules accepted chunks on one AudioContext clock, and exposes the result through an analyser node. Scheduling on one clock avoids the gaps and overlaps that appear when every chunk starts its own source at the current time.

conversation.setVolume(0.8);      // Linear gain from 0 to 1.conversation.outputGain;         // Reactive gain value.conversation.getOutputVolume();  // Latest normalized playback level.

outputVolume controls the initial gain. setVolume clamps values and can accept either a number or an object with a volume property. The value is applied to the playback gain node immediately.

Barge-in

Barge-in is handled locally and confirmed by the server. Two signals can flush playback:

• Local voice activity. When the microphone level crosses vadThreshold (default 0.075) and interruptOnUserSpeech is true, queued audio is flushed with a short gain ramp.
• Server interruption. An interruption event flushes every queued source and updates isSpeaking.

The controller records one interruption when both signals describe the same barge-in. interruptions is the session counter. Set interruptOnUserSpeech: false when you want the agent to finish its current response before listening again.

Devices

Device selection requires an active voice session. Labels are usually hidden until the user grants microphone permission, so refresh the list after the first session starts.

const devices = await conversation.listAudioDevices();const microphones = devices.filter((device) => device.kind === 'audioinput');await conversation.changeInputDevice(microphones[0]?.deviceId);await conversation.changeOutputDevice('speaker-device-id');

Both switch methods return a boolean. A failed switch restores the previous device and publishes a typed error such as device-not-found, device-busy, or device-constraints. Output routing depends on setSinkId support in the browser.

Passing undefined selects the system default. The chosen ids are available on inputDeviceId and outputDeviceId.

Analyser data

The audio manager keeps a 512-bin analyser for the microphone and another for agent playback. Both are safe to call when no session is active and return zeroed data.

const bins = new Uint8Array(512);function paint(): void {  conversation.getInputFrequencyData(bins);  conversation.getOutputFrequencyData(bins);  // Draw the bytes with canvas, SVG, or CSS.}

Pass a reusable buffer to avoid allocating on every animation frame. The buffer must hold at least 512 entries. getInputByteFrequencyData and getOutputByteFrequencyData are aliases for the same data. getInputVolume() and getOutputVolume() return normalized levels when you only need a meter.

AudioVisualizer already handles the animation loop and prefers-reduced-motion. Use the raw analyser methods only when you need a custom visual.

Browser requirements

Voice sessions need a secure context, navigator.mediaDevices.getUserMedia, Web Audio, and WebSocket. An AudioWorklet is preferred, with a ScriptProcessor fallback for older engines.

If a browser cannot satisfy these requirements, call AudioManager.isSupported() before showing a start button, or offer the text only path. See Troubleshooting for the error codes that map to each failure.