Speech to Speech API

/v1/live/sessions follows the primary WebSocket of the OpenAI Live API, with delegation. This page is the wire protocol. For access and a first call, start with the Speech to speech overview. The limits page has rate limits and usage, and Coming from OpenAI Live lists what differs.

Connect

GET wss://api.kugelaudio.com/v1/live/sessions
Authorization: Bearer YOUR_API_KEY

Hosts are in the overview. A client that cannot set the header can send the WebSocket subprotocols openai-live and openai-insecure-api-key.YOUR_API_KEY instead. A browser would expose the key this way, so use it only on your own server.

Every frame is a JSON text frame, and audio is base64 inside it. Without the OpenAI SDK, a whole session looks like this, using the websockets package:

import asyncio
import json
import os

from websockets.asyncio.client import connect


async def main() -> None:
    async with connect(
        "wss://api.kugelaudio.com/v1/live/sessions",
        additional_headers={"Authorization": f"Bearer {os.environ['KUGELAUDIO_API_KEY']}"},
    ) as socket:
        await socket.send(json.dumps({"type": "session.start", "session": {"model": "fluid-1"}}))
        async for frame in socket:
            event = json.loads(frame)
            print(event["type"])
            if event["type"] == "session.started":
                await socket.send(json.dumps({"type": "session.close"}))
            elif event["type"] == "session.closed":
                break


asyncio.run(main())

Send session.start first. The server answers session.started when the session is ready.

Session start

The session object of session.start:

Sending session.kugel, even empty, also turns on the kugel.* events. session.client is refused: it belongs to WebRTC.

Client events

Every client event takes an optional event_id of up to 512 characters, returned as client_event_id on the event that answers it.

content is at most 500 tokens. delegation_id is required but may be null, and a value must be an id from session.delegation.created in this session. You can send results for the same id more than once.

Server events

Transcript and media times are counted from audio samples. They are estimates, not word-aligned timestamps. On a responses delegation you send no commentary: the agent says the model's text.

Kugel events

These arrive only on a session that sent session.kugel. Every event has an event_id.

Errors

An error event has error.code, error.message, error.type (invalid_request_error or server_error) and, where they apply, error.param and error.client_event_id. Errors that end the session say so below, and the rest leave it open.

Refused upgrades

Some refusals come before the upgrade, as an HTTP response and not a close code. The OpenAI SDK and the websockets package raise websockets.exceptions.InvalidStatus and expose it on .response.

Close codes