Realtime voice agent

The Realtime protocol exposes one public speech-to-speech model: kugel-agent-1. You choose instructions, function tools, a numeric voice identifier, and the audio wire format. Recognition, turn detection, reasoning, speech planning, and synthesis providers remain server-managed.

The standard TTS ingress process does not mount these routes, and this repository does not ship a combined production image. An operator must compose the agent behind the authenticated Realtime ingress before these endpoints are available. The standalone agent socket is an unauthenticated, loopback-only debugging surface and is disabled unless explicitly enabled.

POST /v1/realtime/client_secrets
WSS /v1/realtime?model=kugel-agent-1

Authenticate

Server applications can connect with their project API key:

from websockets.sync.client import connect

with connect(
    "wss://api.kugelaudio.com/v1/realtime?model=kugel-agent-1",
    additional_headers={"Authorization": "Bearer YOUR_API_KEY"},
) as socket:
    ...

Do not expose an API key in browser code. Exchange it on your server for a five-minute, project-scoped client secret:

curl -X POST "https://api.kugelaudio.com/v1/realtime/client_secrets" \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "value": "SHORT_LIVED_CLIENT_SECRET",
  "expires_at": 1784811900
}

The browser then connects with the returned value:

const url = new URL("wss://api.kugelaudio.com/v1/realtime");
url.searchParams.set("model", "kugel-agent-1");
url.searchParams.set("client_secret", secretFromYourServer);
const socket = new WebSocket(url);

See Authentication for API-key handling.

Configure a session

The server first sends session.created. Select a numeric voice identifier before requesting output:

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "model": "kugel-agent-1",
    "instructions": "You are a concise support agent.",
    "audio": {
      "input": {
        "format": {"type": "audio/pcm", "rate": 16000},
        "transport": "binary"
      },
      "output": {
        "format": {"type": "audio/pcm", "rate": 24000},
        "transport": "binary",
        "voice": "1656"
      }
    },
    "tools": []
  }
}

The server acknowledges an applied update with session.updated. When present, instructions is the instruction character count and tools lists the session’s declared tool names; the server does not echo the full effective configuration. The selected voice is applied to synthesis and becomes immutable when the first output begins. Per-message voice overrides are not supported.

Provider names, recognition models, transcription configuration, and turn detection are not public session fields. Unknown or internal fields fail with an explicit error event.

Send audio

Supported formats are:

Format	Rates	JSON transport	Binary transport
PCM16 little-endian (`audio/pcm`)	16 or 24 kHz	Base64	Raw bytes
G.711 μ-law (`audio/pcmu`)	8 kHz	Base64	Raw bytes

For JSON transport, send:

{
  "type": "input_audio_buffer.append",
  "audio": "BASE64_AUDIO"
}

For binary transport, send the audio bytes as a binary WebSocket frame. The server emits input_audio_buffer.speech_started and input_audio_buffer.speech_stopped and automatically commits a detected speech boundary. A client may also explicitly commit the current buffer:

{"type": "input_audio_buffer.commit"}

Use input_audio_buffer.clear to discard buffered, uncommitted audio. Clearing input does not cancel an active response; use response.cancel for that. Each commit produces input_audio_buffer.committed, transcription updates, and one completed user conversation item through the deployment’s private perception runtime. Recognition candidates, confidence values, provider versions, endpoint scores, and endpointing controls remain server-side.

Request an agent response

After a committed user turn, request model-authored speech with:

{"type": "response.create"}

The private response runtime receives the latest committed transcript, instructions, and configured tools. A committed user turn is required. Only a Speaker-authored clause associated with that turn enters the serialized TTS and playout path; private reasoning is not sent directly to synthesis. Per-response instructions and tools are not supported in this version. Supplying response configuration returns a not_implemented error and does not start a response:

{
  "type": "response.create",
  "response": {
    "instructions": "Answer in one sentence.",
    "tools": []
  }
}

When the Thinker selects a configured function, the server emits a function_call conversation item followed by response.function_call_arguments.done. Return the exact call_id:

{
  "type": "conversation.item.create",
  "item": {
    "type": "function_call_output",
    "call_id": "call-1",
    "output": "{\"status\":\"shipped\"}"
  }
}

The runtime rejects unknown, duplicate, malformed, and reused call IDs. A valid result continues the private Thinker/Speaker response lifecycle. To report that client-side tool execution failed, return a JSON object whose only field is a non-empty string named error, for example "{\"error\":\"user not found\"}". The runtime exposes that as a failed tool outcome rather than a successful result, allowing the agent to recover without claiming that the operation succeeded. For identity-bearing arguments such as account IDs, declare identity_arguments alongside the JSON schema:

{
  "type": "function",
  "name": "lookup_account",
  "parameters": {
    "type": "object",
    "properties": {
      "account_id": {"type": "string"}
    },
    "required": ["account_id"]
  },
  "identity_arguments": ["account_id"]
}

Every identity argument must name a declared property. If recognition still has plausible spellings for one of these values, the call is rejected rather than dispatching an ambiguous identity to the client tool.

Speak exact fixed text

A force_message speaks client-authored text directly. Sending the item is the entire request; do not follow it with response.create.

{
  "type": "conversation.item.create",
  "item": {
    "type": "force_message",
    "role": "assistant",
    "interruptible": false,
    "content": [
      {
        "type": "output_text",
        "text": "This call may be recorded for quality purposes."
      }
    ]
  }
}

The text is sent to synthesis exactly as supplied after JSON decoding. V1 accepts exactly one nonempty output_text part with at most 2,000 Unicode code points. It does not accept SSML, audio parts, tool calls, or per-message voices. Fixed speech uses the same serialized output floor as agent-authored speech. If interruptible is false, caller audio received during playback is discarded and the server emits kugel.input_audio.suppressed.

Response lifecycle

A successful audio response uses this lifecycle:

response.created
conversation.item.created (agent turn only)
response.output_audio_transcript.delta
response.output_audio.delta or a binary audio frame
response.output_audio_transcript.done
client: response.output_audio.played
response.output_audio.done
response.done

With base64 output, each response.output_audio.delta contains encoded audio. With binary output, audio chunks are raw binary frames and lifecycle metadata remains JSON. The final transcript for a fixed message equals its supplied text. response.done is emitted only after the client reports a terminal played span; sending bytes to a socket is not treated as audible playout. Audio deltas are emitted incrementally as synthesis produces them. response.output_audio_transcript.done is ordered behind the final audio frame for its clause. Clients should not send a terminal playout acknowledgement until they have received this marker and actually played the scheduled audio. When final reasoning takes longer than the Speaker’s bridge decision, one response can contain two separately identified clauses: a short Speaker-authored bridge followed by the substantive answer. After playing audio, report the exact cumulative sample and Unicode-code-point cut:

{
  "type": "response.output_audio.played",
  "response_id": "response-1",
  "item_id": "item-1",
  "audio_end_sample": 24000,
  "text_end_offset": 48,
  "played_through_ms": 1000,
  "completed": true
}

Set completed only when the full text and produced audio were played. Partial acknowledgements update history with the exact prefix but do not release the output floor. When caller speech interrupts an interruptible response, the server estimates the heard prefix from emitted samples and synthesis word timestamps, emits response.output_audio.done for that prefix, and then emits response.cancelled. Do not acknowledge a response after it is cancelled. Offsets that regress, exceed produced output, or identify another item fail explicitly. Cancel the active or named interruptible response with:

{
  "type": "response.cancel",
  "response_id": "response-1"
}

The server emits response.cancelled after recording its best estimate of the heard prefix. Speech beyond that prefix is not added to conversation history. conversation.item.truncate is reserved by the schema but is not implemented by the current shared runtime.

Client events

session.update
input_audio_buffer.append
input_audio_buffer.commit
input_audio_buffer.clear
conversation.item.create
response.create
response.cancel
response.output_audio.played

Server events

session.created
session.updated
input_audio_buffer.speech_started
input_audio_buffer.speech_stopped
input_audio_buffer.committed
conversation.item.created
conversation.item.input_audio_transcription.updated
response.created
response.function_call_arguments.done
response.output_audio.delta
response.output_audio.done
response.output_audio.unavailable
response.output_audio_transcript.delta
response.output_audio_transcript.done
response.done
response.cancelled
kugel.input_audio.suppressed
error

Client-driven conversation.item.truncate is schema-reserved but is not handled by the current shared runtime.

Get started

Build a voice agent

Generate audio

Voices

Deploy

SDK Reference

API Reference

Realtime voice agent

Authenticate

Configure a session

Send audio

Request an agent response

Speak exact fixed text

Response lifecycle

Client events

Server events

​Authenticate

​Configure a session

​Send audio

​Request an agent response

​Speak exact fixed text

​Response lifecycle

​Client events

​Server events

Authenticate

Configure a session

Send audio

Request an agent response

Speak exact fixed text

Response lifecycle

Client events

Server events