> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kugelaudio.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Realtime voice agent

> Configure and stream a Kugel voice-agent session over one WebSocket.

The Realtime protocol exposes one public speech-to-speech model:
`kugel-agent-1`. You choose instructions, function tools, a numeric voice
identifier, and the audio wire format. Recognition, turn detection, reasoning,
speech planning, and synthesis providers remain server-managed.

<Warning>
  The standard TTS ingress process does not mount these routes, and this
  repository does not ship a combined production image. An operator must
  compose the agent behind the authenticated Realtime ingress before these
  endpoints are available. The standalone agent socket is an unauthenticated,
  loopback-only debugging surface and is disabled unless explicitly enabled.
</Warning>

```text theme={null}
POST /v1/realtime/client_secrets
WSS /v1/realtime?model=kugel-agent-1
```

## Authenticate

Server applications can connect with their project API key:

```python theme={null}
from websockets.sync.client import connect

with connect(
    "wss://api.kugelaudio.com/v1/realtime?model=kugel-agent-1",
    additional_headers={"Authorization": "Bearer YOUR_API_KEY"},
) as socket:
    ...
```

Do not expose an API key in browser code. Exchange it on your server for a
five-minute, project-scoped client secret:

```bash theme={null}
curl -X POST "https://api.kugelaudio.com/v1/realtime/client_secrets" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

```json theme={null}
{
  "value": "SHORT_LIVED_CLIENT_SECRET",
  "expires_at": 1784811900
}
```

The browser then connects with the returned value:

```javascript theme={null}
const url = new URL("wss://api.kugelaudio.com/v1/realtime");
url.searchParams.set("model", "kugel-agent-1");
url.searchParams.set("client_secret", secretFromYourServer);
const socket = new WebSocket(url);
```

See [Authentication](/api-reference/authentication) for API-key handling.

## Configure a session

The server first sends `session.created`. Select a numeric voice identifier
before requesting output:

```json theme={null}
{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "model": "kugel-agent-1",
    "instructions": "You are a concise support agent.",
    "audio": {
      "input": {
        "format": {"type": "audio/pcm", "rate": 16000},
        "transport": "binary"
      },
      "output": {
        "format": {"type": "audio/pcm", "rate": 24000},
        "transport": "binary",
        "voice": "1656"
      }
    },
    "tools": []
  }
}
```

The server acknowledges an applied update with `session.updated`. When present,
`instructions` is the instruction character count and `tools` lists the
session's declared tool names; the server does not echo the full effective
configuration. The selected voice is applied to synthesis and becomes immutable
when the first output begins. Per-message voice overrides are not supported.

<Warning>
  Provider names, recognition models, transcription configuration, and turn
  detection are not public session fields. Unknown or internal fields fail
  with an explicit `error` event.
</Warning>

## Send audio

Supported formats are:

| Format                            | Rates        | JSON transport | Binary transport |
| --------------------------------- | ------------ | -------------- | ---------------- |
| PCM16 little-endian (`audio/pcm`) | 16 or 24 kHz | Base64         | Raw bytes        |
| G.711 μ-law (`audio/pcmu`)        | 8 kHz        | Base64         | Raw bytes        |

For JSON transport, send:

```json theme={null}
{
  "type": "input_audio_buffer.append",
  "audio": "BASE64_AUDIO"
}
```

For binary transport, send the audio bytes as a binary WebSocket frame.
The server emits `input_audio_buffer.speech_started` and
`input_audio_buffer.speech_stopped` and automatically commits a detected
speech boundary. A client may also explicitly commit the current buffer:

```json theme={null}
{"type": "input_audio_buffer.commit"}
```

Use `input_audio_buffer.clear` to discard buffered, uncommitted audio. Clearing
input does not cancel an active response; use `response.cancel` for that.

Each commit produces `input_audio_buffer.committed`, transcription updates,
and one completed user conversation item through the deployment's private
perception runtime. Recognition candidates, confidence values, provider
versions, endpoint scores, and endpointing controls remain server-side.

## Request an agent response

After a committed user turn, request model-authored speech with:

```json theme={null}
{"type": "response.create"}
```

The private response runtime receives the latest committed transcript,
instructions, and configured tools. A committed user turn is required. Only a
Speaker-authored clause associated with that turn enters the serialized TTS and
playout path; private reasoning is not sent directly to synthesis.

Per-response instructions and tools are not supported in this version. Supplying
`response` configuration returns a `not_implemented` error and does not start a
response:

```json theme={null}
{
  "type": "response.create",
  "response": {
    "instructions": "Answer in one sentence.",
    "tools": []
  }
}
```

When the Thinker selects a configured function, the server emits a
`function_call` conversation item followed by
`response.function_call_arguments.done`. Return the exact `call_id`:

```json theme={null}
{
  "type": "conversation.item.create",
  "item": {
    "type": "function_call_output",
    "call_id": "call-1",
    "output": "{\"status\":\"shipped\"}"
  }
}
```

The runtime rejects unknown, duplicate, malformed, and reused call IDs. A
valid result continues the private Thinker/Speaker response lifecycle.
To report that client-side tool execution failed, return a JSON object whose
only field is a non-empty string named `error`, for example
`"{\"error\":\"user not found\"}"`. The runtime exposes that as a failed tool
outcome rather than a successful result, allowing the agent to recover without
claiming that the operation succeeded.

For identity-bearing arguments such as account IDs, declare
`identity_arguments` alongside the JSON schema:

```json theme={null}
{
  "type": "function",
  "name": "lookup_account",
  "parameters": {
    "type": "object",
    "properties": {
      "account_id": {"type": "string"}
    },
    "required": ["account_id"]
  },
  "identity_arguments": ["account_id"]
}
```

Every identity argument must name a declared property. If recognition still has
plausible spellings for one of these values, the call is rejected rather than
dispatching an ambiguous identity to the client tool.

## Speak exact fixed text

A `force_message` speaks client-authored text directly. Sending the item is
the entire request; do not follow it with `response.create`.

```json theme={null}
{
  "type": "conversation.item.create",
  "item": {
    "type": "force_message",
    "role": "assistant",
    "interruptible": false,
    "content": [
      {
        "type": "output_text",
        "text": "This call may be recorded for quality purposes."
      }
    ]
  }
}
```

The text is sent to synthesis exactly as supplied after JSON decoding. V1
accepts exactly one nonempty `output_text` part with at most 2,000 Unicode
code points. It does not accept SSML, audio parts, tool calls, or
per-message voices.

Fixed speech uses the same serialized output floor as agent-authored speech.
If `interruptible` is `false`, caller audio received during playback is
discarded and the server emits `kugel.input_audio.suppressed`.

## Response lifecycle

A successful audio response uses this lifecycle:

```text theme={null}
response.created
conversation.item.created (agent turn only)
response.output_audio_transcript.delta
response.output_audio.delta or a binary audio frame
response.output_audio_transcript.done
client: response.output_audio.played
response.output_audio.done
response.done
```

With base64 output, each `response.output_audio.delta` contains encoded audio.
With binary output, audio chunks are raw binary frames and lifecycle metadata
remains JSON. The final transcript for a fixed message equals its supplied
text. `response.done` is emitted only after the client reports a terminal
played span; sending bytes to a socket is not treated as audible playout.
Audio deltas are emitted incrementally as synthesis produces them.
`response.output_audio_transcript.done` is ordered behind the final audio frame
for its clause. Clients should not send a terminal playout acknowledgement
until they have received this marker and actually played the scheduled audio.

When final reasoning takes longer than the Speaker's bridge decision, one
response can contain two separately identified clauses: a short
Speaker-authored bridge followed by the substantive answer.

After playing audio, report the exact cumulative sample and Unicode-code-point
cut:

```json theme={null}
{
  "type": "response.output_audio.played",
  "response_id": "response-1",
  "item_id": "item-1",
  "audio_end_sample": 24000,
  "text_end_offset": 48,
  "played_through_ms": 1000,
  "completed": true
}
```

Set `completed` only when the full text and produced audio were played. Partial
acknowledgements update history with the exact prefix but do not release the
output floor. When caller speech interrupts an interruptible response, the
server estimates the heard prefix from emitted samples and synthesis word
timestamps, emits `response.output_audio.done` for that prefix, and then emits
`response.cancelled`. Do not acknowledge a response after it is cancelled.
Offsets that regress, exceed produced output, or identify another item fail
explicitly.

Cancel the active or named interruptible response with:

```json theme={null}
{
  "type": "response.cancel",
  "response_id": "response-1"
}
```

The server emits `response.cancelled` after recording its best estimate of the
heard prefix. Speech beyond that prefix is not added to conversation history.
`conversation.item.truncate` is reserved by the schema but is not implemented
by the current shared runtime.

## Client events

```text theme={null}
session.update
input_audio_buffer.append
input_audio_buffer.commit
input_audio_buffer.clear
conversation.item.create
response.create
response.cancel
response.output_audio.played
```

## Server events

```text theme={null}
session.created
session.updated
input_audio_buffer.speech_started
input_audio_buffer.speech_stopped
input_audio_buffer.committed
conversation.item.created
conversation.item.input_audio_transcription.updated
response.created
response.function_call_arguments.done
response.output_audio.delta
response.output_audio.done
response.output_audio.unavailable
response.output_audio_transcript.delta
response.output_audio_transcript.done
response.done
response.cancelled
kugel.input_audio.suppressed
error
```

Client-driven `conversation.item.truncate` is schema-reserved but is not
handled by the current shared runtime.