Fit it into your stack
See where speech to speech sits in your system, and how a caller's audio reaches the agent: from a phone line, a web page, LiveKit or your own code.
On this page
The parts of a call
| Part | What it does | Who runs it |
|---|---|---|
| The caller's connection | Carries the caller's audio: a phone line, a web page or an app | You, or your phone provider |
| Your server | Starts the session, streams the caller's audio in and the agent's audio out | You |
| Fluid 1 | Listens, decides when to answer, speaks, and delegates what it cannot answer | KugelAudio |
| Back office | Does the work a turn needs and sends the result | You, or KugelAudio |
The delegation travels on the same WebSocket as the audio, so a call needs one connection, not two.
Keep your key on your server
The WebSocket takes your API key. Open it from your server, never from a web page or an app: anyone who loads the page can read a key that is in it. The caller's device talks to your server, and your server talks to KugelAudio.
Pick how the audio reaches the agent
| The caller is on | How the audio gets there | Audio format |
|---|---|---|
| A phone line (Twilio, Telnyx, a SIP gateway) | Your phone provider streams the call to your server, which passes each frame on | audio/pcmu or audio/pcma at 8 kHz, both ways, unchanged |
| A web page or an app | The page or app streams its microphone to your server, which passes it on | audio/pcm at 24 or 16 kHz |
| LiveKit Agents | LiveKit's OpenAI plugin holds the WebSocket for you | Chosen by the plugin |
| Your own microphone, to try it | A script streams the microphone and plays the reply: the quickstart | audio/pcm at 24 kHz |
A phone line
Phone systems send 8 kHz G.711, and the agent answers in the same format, so your server forwards frames without converting them. Ask for the format when you start the session: Connect a phone line.
A web page or an app
Your page sends the microphone to your server over a connection you already have, a WebSocket for example, and plays what comes back. Your server decides who may talk to the agent, so your own sign-in and rate limits apply.
LiveKit Agents
LiveKit's OpenAI plugin connects to speech to speech in place of your own WebSocket code. See speech to speech in LiveKit.
Build it to last
- One session per call. A session is one conversation on one socket, lasts at most an hour and is not stored. Open a new one for every call. See limits.
- Start the session before the caller speaks. The model, the instructions and the audio format are fixed at the start. You can add to the instructions during the call.
- Stream audio as it arrives. Send the caller's audio in small frames as you receive it, and play the agent's audio as it comes. Do not wait for a whole sentence.
- Stop playback when the caller interrupts. The agent stops talking when the caller cuts in. If you buffer its audio, drop the queue.
- Answer delegations quickly. The caller waits for your result. For slow work, report progress so the agent can tell them.
- Plan for a dropped connection. A broken socket ends the session. Start a new one with what you kept, and the agent carries on.
- Retry a refused session with backoff. See retry a refused session.