> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kugelaudio.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Speech to Speech API

> WebSocket reference for speech to speech with fluid-1: connecting, session fields, client and server events, limits, errors and close codes.

`/v1/live/sessions` follows the primary WebSocket of the
[OpenAI Live API](https://developers.openai.com/api/docs/guides/voice-websockets?api=live), with
[delegation](https://developers.openai.com/api/docs/guides/live-delegation). This page is the wire
protocol. For access and a first call, start with the
[Speech to speech overview](/speech-to-speech/overview). The
[limits page](/speech-to-speech/limits) has rate limits and usage, and
[Coming from OpenAI Live](/speech-to-speech/openai-compatibility) lists what differs.

<Info>
  For how an event is meant to be used, read [OpenAI's Live documentation](/speech-to-speech/overview#use-openais-documentation).
  It applies as written, except for what [Coming from OpenAI Live](/speech-to-speech/openai-compatibility) lists.
</Info>

## Connect

```http
GET wss://api.kugelaudio.com/v1/live/sessions
Authorization: Bearer YOUR_API_KEY
```

Hosts are in the [overview](/speech-to-speech/overview#choose-a-host). A client that cannot
set the header can send the WebSocket subprotocols `openai-live` and
`openai-insecure-api-key.YOUR_API_KEY` instead. A browser would expose the key this way,
so use it only on your own server.

Every frame is a JSON text frame, and audio is base64 inside it. Without the OpenAI SDK,
a whole session looks like this, using the `websockets` package:

```python raw_session.py
import asyncio
import json
import os

from websockets.asyncio.client import connect


async def main() -> None:
    async with connect(
        "wss://api.kugelaudio.com/v1/live/sessions",
        additional_headers={"Authorization": f"Bearer {os.environ['KUGELAUDIO_API_KEY']}"},
    ) as socket:
        await socket.send(json.dumps({"type": "session.start", "session": {"model": "fluid-1"}}))
        async for frame in socket:
            event = json.loads(frame)
            print(event["type"])
            if event["type"] == "session.started":
                await socket.send(json.dumps({"type": "session.close"}))
            elif event["type"] == "session.closed":
                break


asyncio.run(main())
```

Send `session.start` first. The server answers `session.started` when the session is
ready.

## Session start

The `session` object of `session.start`:

<ParamField body="model" type="string" required>
  `fluid-1`, or its alias `gpt-live-1`. Anything else fails with `unsupported_model`.
</ParamField>

<ParamField body="instructions" type="string">
  How the agent behaves. At most 16,384 tokens, counted with the `o200k_base` encoding.
</ParamField>

<ParamField body="audio.format" type="object" default='{"type": "audio/pcm", "rate": 24000}'>
  The format in both directions: `audio/pcm` at `rate` 16000 or 24000 (PCM16,
  little-endian, mono), or `audio/pcmu` or `audio/pcma` at `rate` 8000.
</ParamField>

<ParamField body="audio.output.voice" type="string | object">
  A numeric voice id as a string, or `{"id": "973"}`. Another name selects the default
  voice and sends an `info` event with the code `kugel_voice_alias`.
</ParamField>

<ParamField body="delegation" type="object" default='{"type": "client"}'>
  `{"type": "client"}`, where your backend does the delegated work, or
  `{"type": "responses", "responses": {...}}`, where the service does and your client
  runs the tools. It cannot change after the start.
</ParamField>

<ParamField body="delegation.responses.model" type="string">
  `gpt-6.1-sol`, `gpt-6-astra` or `gpt-6.1-terra`. Left out, the service chooses. The
  guide says [where these run](/speech-to-speech/back-office#choose-the-model).
</ParamField>

<ParamField body="delegation.responses" type="object">
  Also takes `instructions`, `tools` (function tools only), `tool_choice` (`auto`, `none`
  or `required`), `parallel_tool_calls` and `reasoning.effort`. Anything else fails with
  `invalid_event`.
</ParamField>

<ParamField body="input" type="array">
  Earlier conversation: at most 128 messages and 8,192 tokens. A message has a `role`
  (`developer`, `user` or `assistant`) and one text part in `content`. The agent skips
  its greeting when `user` or `assistant` messages are present. A `developer` message
  joins the instructions.
</ParamField>

<ParamField body="store" type="boolean" default="false">
  `true` fails: sessions are not stored.
</ParamField>

<ParamField body="kugel.tools" type="array">
  What your backend can do, at most 64 function tools with `name`, `description` and
  `parameters`. The agent delegates a turn when answering it needs one of them. Left
  out, it delegates the work your instructions list for a back office or tools, else
  anything it needs your backend for. `[]` means it never delegates.
</ParamField>

<ParamField body="kugel.language" type="string">
  The locale the call starts in, such as `de-DE`. A language the service cannot hold a
  call in ends the session with an error.
</ParamField>

<ParamField body="kugel.quick_answers" type="boolean">
  Whether the agent answers as soon as the caller's turn is over. Left out, the service
  default applies.
</ParamField>

Sending `session.kugel`, even empty, also turns on the [`kugel.*` events](#kugel-events).
`session.client` is refused: it belongs to WebRTC.

## Client events

Every client event takes an optional `event_id` of up to 512 characters, returned as
`client_event_id` on the event that answers it.

| Event | Fields | What it does |
|---|---|---|
| `session.start` | `session` | Starts the session. Allowed once. |
| `session.update` | `session` | Only `{}` or `{"delegation": {"type": "client"}}` is accepted, answered with `session.updated`. The model, instructions, audio and input cannot change after the start. |
| `session.input_audio.append` | `audio` | Base64 audio in the session's format. PCM16 needs whole 16-bit samples. |
| `session.input_audio.mute` | | The service hears silence until you unmute, so media time keeps running. It does not cancel the agent's speech. Answered with `session.input_audio.muted`. |
| `session.input_audio.unmute` | | Ends the mute. Answered with `session.input_audio.unmuted`. |
| `session.commentary.append` | `content`, `delegation_id`, `expects_reply` | Facts the agent can say in its own words. With a `delegation_id`, the result of that delegation: the agent says it as its reply. With `null`, its own line: the agent says it word for word, or follows it when it is an instruction to the agent ("Tell the caller …"). To set the greeting, put it in `instructions`. `Nothing to add.` and a result that only acknowledges (`Got it.`) are said by nobody. With `expects_reply: true` it says `content` word for word, and the caller's next finished turn opens a new delegation. A `null` line still waiting to be said when the caller finishes a turn is dropped, because that turn may have answered it. |
| `session.thinking.append` | `content`, `delegation_id` | Silent context for a delegation in progress. Give the delegation's id. |
| `session.instructions.append` | `content`, `delegation_id` | A change to how the agent behaves. |
| `response.item.create` | `item` | `responses` delegation: the result of one tool call, `{"type": "function_call_output", "call_id": "...", "output": "..."}`. |
| `response.create` | | `responses` delegation: send once, after every call of a response has its result. |
| `session.close` | | Finishes the reply in progress, then ends the session. |

`content` is at most 500 tokens. `delegation_id` is required but may be `null`, and a
value must be an id from `session.delegation.created` in this session. You can send
results for the same id more than once.

## Server events

| Event | Fields | Meaning |
|---|---|---|
| `session.started` | `session` | The session is ready. `session` has `id`, `expires_at`, `model`, `status`, `audio`, `delegation`, `instructions`, `store` and `input`. |
| `session.updated` | `session` | Answer to `session.update`. |
| `session.output_audio.delta` | `delta` | Base64 audio of the agent, in the session's format. |
| `session.input_transcript.delta` | `delta`, `start_ms`, `end_ms` | What the caller says, streamed while they speak: append each `delta` in order. The first of a turn arrives with the caller's first words and starts where they began to speak; each word comes once. Words already sent are not taken back when the final transcript differs. |
| `session.output_transcript.delta` | `delta`, `start_ms`, `end_ms` | What the agent says, a few words at a time as its audio is sent: append each `delta` in order. `start_ms` and `end_ms` span the audio sent with it, so a reply sorts after the caller turn it answers. A reply cut off ends with the words its audio reached. |
| `session.delegation.created` | `delegation`, `offset_ms` | A turn needs your backend. `delegation` has `id`, `target` (`client` or `responses`) and, for `responses`, `response_id`. It follows the caller turn it was opened on, with that turn's final transcript. |
| `session.commentary.appended`, `session.thinking.appended`, `session.instructions.appended` | `start_ms`, `end_ms` | The context you sent has taken effect. |
| `session.input_audio.muted`, `session.input_audio.unmuted` | | Answers to mute and unmute. |
| `response.event` | `delegation_id`, `event` | `responses` delegation: one step of the back office. `event` has the `type` `response.created`, `response.output_item.done` (with a function call in `item`), `response.completed` or `response.failed` (with `response.error`). |
| `session.usage.updated` | `usage.seconds` | Seconds of usage so far, sent when a reply is done. |
| `info` | `code`, `message` | A note, not a failure. |
| `error` | `error` | See [Errors](#errors). |
| `session.closed` | `reason`, `session`, `usage.seconds` | The session ended. `reason` is `close_requested`, `expired`, `content`, `remote_hangup` or `connection_lost`. |

Transcript and media times are counted from audio samples. They are estimates, not
word-aligned timestamps. On a `responses` delegation you send no commentary: the agent
says the model's text.

### Kugel events

These arrive only on a session that sent `session.kugel`. Every event has an `event_id`.

| Event | Fields | Meaning |
|---|---|---|
| `kugel.response.created` | `response_id` | The agent starts a reply. |
| `kugel.response.done` | `response_id` | The reply is finished. |
| `kugel.response.cancelled` | `response_id` | The reply stopped, for example because the caller spoke. Clear the audio you have not played. |
| `kugel.input_audio.speech_started`, `kugel.input_audio.speech_stopped` | `at` | The caller started or stopped speaking. |
| `kugel.input_transcript.partial` | `text` | The caller's line so far. Each one replaces the last, and `""` clears it. The final text is `kugel.input_transcript.committed`. |
| `kugel.input_transcript.committed` | `turn_id`, `text` | One caller turn, whole, as the agent understood it, including corrections to words already streamed. The same `turn_id` again replaces that turn: the caller went on before the agent answered, and `text` is the whole turn again. |
| `kugel.input_transcript.alternatives` | `text`, `word_alternatives` | After a caller's turn, the words that may be spelled otherwise, each with `alternatives` of `spelling` and `probability`, most likely first. Use them when a lookup with the written spelling finds nothing. |
| `kugel.output_audio.unavailable` | `reason` | Part of the reply has text but no audio. Without `session.kugel` this is an `error` with the code `audio_unavailable`. |

## Errors

An `error` event has `error.code`, `error.message`, `error.type`
(`invalid_request_error` or `server_error`) and, where they apply, `error.param` and
`error.client_event_id`. Errors that end the session say so below, and the rest leave it
open.

| `error.code` | When |
|---|---|
| `invalid_event` | The frame is not a valid client event, or a session field is not accepted. `error.message` says why. |
| `invalid_audio` | The audio is not valid base64, or PCM16 audio has an incomplete sample. |
| `unsupported_model` | `session.model` or `delegation.responses.model` is not offered. |
| `unsupported_language` | The service cannot hold a call in this language. The session ends. |
| `unsupported_responses_delegation` | A `response.*` event on a client-delegation session. |
| `back_office_unavailable` | The `responses` model is not available right now. |
| `session_not_started` | Another event came before `session.start`. |
| `session_already_started` | A second `session.start`. |
| `immutable_field_update` | `session.update` tried to change the delegation. |
| `unknown_delegation` | `delegation_id` is not a delegation of this session. |
| `session_closing` | An event came after `session.close`, while the last reply finishes. |
| `session_closed` | An event came after the session ended. |
| `context_not_applied` | The session ended before context you sent took effect. |
| `response_failed` | A reply failed. |
| `audio_unavailable` | Part of a reply has text but no audio. |
| `tool_output_timeout` | `responses` delegation: no `response.create` within 60 seconds. The caller hears that it failed, and the call goes on. |
| `back_office_failed` | `responses` delegation: the model call failed. The caller hears that it failed, and the call goes on. |
| `rate_limit_exceeded` | The session was refused under the [rate limits](/api-reference/errors#rate-limits) or for lack of capacity. Close code `1013` follows. Open a new session after a pause. |
| `billing_limit_reached` | The credits ran out during the call. Close code `1008` follows. |
| `service_unavailable` | A failure on the service's side. Close code `1011` follows, and on a call in progress `session.closed` with the reason `connection_lost`. |

### Refused upgrades

Some refusals come before the upgrade, as an HTTP response and not a close code. The
OpenAI SDK and the `websockets` package raise `websockets.exceptions.InvalidStatus` and
expose it on `.response`.

| Status | Body | Meaning |
|---|---|---|
| `403` | `{"error": "This API key does not have the realtime:write permission", "error_code": "PERMISSION_DENIED", "code": 403}` | The key's [permissions](/api-reference/authentication#key-permissions) do not allow **Realtime sessions**. |
| `403` | `{"error": "Speech-to-speech is not enabled for this organization.", "error_code": "PERMISSION_DENIED", "code": 403}` | Your organization is not enabled yet. |
| `403` | None | The key is missing or invalid, the credits are spent, or they could not be checked. |

### Close codes

| Code | Meaning |
|---|---|
| `1000` | The session ended normally, or reached its one-hour limit. |
| `1008` | The credits ran out during the call, after a `billing_limit_reached` error. |
| `1011` | A failure on the service's side, after a `service_unavailable` error. |
| `1013` | Try again later: the session was refused after a `rate_limit_exceeded` error, or the server is restarting, with no error first. Open a new session after a pause. |