Speech to speech
Give your product a voice agent that hears callers, answers in a natural voice and checks your systems when it needs to, over one WebSocket.
On this page
Speech to speech runs on Kugel Fluid 1 (fluid-1). To see where it sits in your system before
you write code, read Fit it into your stack.
What you get
| Feature | What it does |
|---|---|
| Audio both ways | PCM16 at 16 or 24 kHz, or G.711 (audio/pcmu, audio/pcma) at 8 kHz. The agent answers in the format you choose. |
| Transcripts | Both sides of the call arrive as text next to the audio. |
| Your systems in the call | The agent asks your backend when a turn needs it, and tells the caller the result. |
| Interruptions | The caller can talk over the agent, and an event tells you to stop playing audio. |
| OpenAI SDK support | The endpoint follows the OpenAI Live API, so the official OpenAI Python SDK works with a KugelAudio key, and OpenAI's documentation answers your questions. You need no KugelAudio SDK. |
How a call works
- Your server opens one WebSocket and starts a session with instructions for the agent.
- You stream the caller's audio in. The agent's voice streams back, with text of both sides.
- The agent answers what it can from your instructions and the conversation.
- When a turn needs your systems, the agent tells you. This is a delegation.
- You do the work and send the result. The agent tells the caller.
- You close the session, and the event that confirms it reports the seconds used.
Terms
| Term | What it means |
|---|---|
| Session | One conversation on one WebSocket. It lasts at most one hour and is not stored. |
| Agent | The voice the caller talks to. It listens, decides what to do and speaks. |
| Delegation | The agent's request for help with one turn. It tells you when a turn needs your systems. |
| Back office | Whatever does the delegated work: your own backend, or a model KugelAudio runs for you. |
| Commentary | Facts you send for the agent to tell the caller. |
| Thinking | Progress you send for the agent to know, without saying it. |
| Instructions | The agent's brief. You set it at the start and can add to it during the call. |
Choose who does the work
| Your backend | KugelAudio's back office | |
|---|---|---|
delegation.type | client | responses |
| Who decides how to do the work | Your code | A model KugelAudio runs |
| Who runs your tools | Your code | Your code, when the model asks |
| Who writes what the caller hears | The agent, from the facts you send | The model, word for word |
| Choose it when | You already have a backend, or your own agent | You want to list your tools and have a model use them |
Use OpenAI's documentation
Speech to speech follows the OpenAI Live API, and you build with OpenAI's own SDK. These pages cover what is KugelAudio's: access, hosts, voices, the back office and limits. For every other question, OpenAI's documentation applies as written:
| Question | OpenAI's documentation |
|---|---|
| Where do I start? | Getting started with GPT-Live |
| How do I connect, start and close a session, and which audio formats are there? | WebSockets |
| Which events carry audio and transcripts, and how do I mute or add earlier conversation? | Managing sessions |
| How do delegation, commentary, thinking and tools work? | Delegation and tools |
| How do I call it from Python? | OpenAI Python SDK |
Where KugelAudio differs, Coming from OpenAI Live lists it. Everything not listed there works as OpenAI describes.
Prerequisites
- An API key allowed Realtime sessions. In the dashboard this is Realtime sessions set
to Allowed (
realtime:write), or an unrestricted key. See key permissions.
Choose a host
Set the SDK's base_url to one of these, with /v1 at the end. The SDK opens
/v1/live/sessions on it.
| Host | base_url |
|---|---|
| Default | https://api.kugelaudio.com/v1 |
| EU | https://api.eu.kugelaudio.com/v1 |
| US | https://api.us.kugelaudio.com/v1 |
Send the key as the dashboard shows it. The eu- key prefix belongs to the KugelAudio SDKs,
so a prefixed key fails here.