Skip to content
KugelAudioDocs

Connect a browser with WebRTC

Let a web page or an app talk to the agent over WebRTC: your server creates the call with your key, and the audio travels straight between the browser and KugelAudio.

On this page

With WebRTC the caller's browser sends its microphone straight to KugelAudio and plays the agent's audio as it arrives. Your server carries no audio. It creates the call with your API key.

Your server can also follow and steer the call over a second connection.

The call follows the WebRTC part of the OpenAI Live API. Use the WebSocket when the audio already passes through your server, as it does on a phone line.

How a call is set up

Browser
Microphone and speaker
1. offer, 4. answer
Your server
Checks who the user is, holds the key
2. create the call, 3. answer
Fluid 1
Your server creates the call; the audio does not pass through it
Browser
WebRTC: audio and events
Fluid 1
Then the audio and the events go direct
  1. The browser creates a WebRTC offer and sends it to your server.
  2. Your server checks that this user may start a call and posts the offer, with the session settings and your API key, to /v1/live/sessions.
  3. KugelAudio returns an answer and the session id.
  4. Your server passes the answer to the browser. The browser connects, and the agent starts.

Your key never reaches the browser. Anyone who loads a page can read a key that is in it, so the browser cannot create a call on its own.

Create the call on your server

WebRTC calls run in the EU. Create them on api.eu.kugelaudio.com, wherever your server runs: the US host does not take them.

HTTP
POST https://api.eu.kugelaudio.com/v1/live/sessions
Authorization: Bearer YOUR_API_KEY
Content-Type: application/json
JSON
{
  "session": {
    "model": "fluid-1",
    "instructions": "You answer for a parcel service. Keep replies short."
  },
  "transport": { "type": "webrtc", "sdp": "v=0\r\n..." }
}

session takes the fields of session.start, except those listed under what differs. transport.sdp is the browser's offer, unchanged. The answer is 201:

JSON
{
  "session": { "id": "live_3f2a9c1d5b7e4a60812c9d4e6f7a8b90" },
  "transport": { "type": "webrtc", "sdp": "v=0\r\n..." }
}

Give transport.sdp to the browser. session.id names the call.

Keep session.id if your server will attach to the call.

A minimal endpoint, with FastAPI and httpx:

calls.py
import os

import httpx
from fastapi import FastAPI, HTTPException, Request, Response

app = FastAPI()


@app.post("/calls")
async def create_call(request: Request) -> Response:
    # Check here that this user may start a call.
    offer = (await request.body()).decode()
    async with httpx.AsyncClient(timeout=20) as client:
        created = await client.post(
            "https://api.eu.kugelaudio.com/v1/live/sessions",
            headers={"Authorization": f"Bearer {os.environ['KUGELAUDIO_API_KEY']}"},
            json={
                "session": {
                    "model": "fluid-1",
                    "instructions": "You answer for a parcel service. Keep replies short.",
                },
                "transport": {"type": "webrtc", "sdp": offer},
            },
        )
    if created.status_code != 201:
        raise HTTPException(status_code=502, detail=created.json()["error"]["code"])
    call = created.json()
    # call["session"]["id"] names the call.
    return Response(call["transport"]["sdp"], media_type="application/sdp")

Connect from the browser

call.js
const connection = new RTCPeerConnection();

// Play the agent.
const speaker = new Audio();
speaker.autoplay = true;
connection.ontrack = (event) => {
  speaker.srcObject = new MediaStream([event.track]);
};

// Send the microphone.
const microphone = await navigator.mediaDevices.getUserMedia({ audio: true });
connection.addTrack(microphone.getAudioTracks()[0], microphone);

// Events travel on a data channel with this name.
const events = connection.createDataChannel("oai-events");
events.onmessage = (message) => {
  const event = JSON.parse(message.data);
  console.log(event.type);
};

await connection.setLocalDescription(await connection.createOffer());
const created = await fetch("/calls", {
  method: "POST",
  headers: { "Content-Type": "application/sdp" },
  body: connection.localDescription.sdp,
});
await connection.setRemoteDescription({ type: "answer", sdp: await created.text() });

The browser needs no STUN or TURN server, so RTCPeerConnection takes no iceServers. Call getUserMedia after a click or a tap: browsers ask for the microphone, and most refuse to play audio on a page the user has not touched.

The session starts when the connection is up, so the agent's greeting is not spoken before anyone can hear it. To hang up, call connection.close().

Events on the data channel

Every server event except the audio arrives on oai-events as JSON text, and the page can send client events the same way:

JavaScript
events.send(JSON.stringify({ type: "session.input_audio.mute" }));

The page is in the caller's hands, so do not let it decide what the agent is told.

Answer delegations and change the instructions from your server, over the sideband.

What differs from the WebSocket

TopicOver WebRTC
Starting the sessionThe create call starts it. A session.start on the data channel is answered with an error event.
AudioOpus on the media tracks, both ways. session.input_audio.append is answered with an error event, and no session.output_audio.delta arrives.
session.audio.formatSet by the transport. Sending it is refused with unsupported_on_webrtc.
session.kugel and the kugel.* eventsNot available. Sending session.kugel is refused with unsupported_on_webrtc, so the call starts in the language of your instructions.
session.clientNot available. Refused with unsupported_on_webrtc.
InterruptionsNothing to do. The audio arrives at the pace of speech, and the agent's speech stops in the stream when the caller cuts in.
Limits and usageThe same as a WebSocket session. A call is one session.

When a call ends or the connection breaks

A call lives on one connection. It cannot move to another network path or be resumed, so when connection.connectionState becomes failed or closed, or session.closed arrives, the call is over. To carry on, create a new call and pass what was said in input.

JavaScript
connection.onconnectionstatechange = () => {
  if (connection.connectionState === "failed") {
    connection.close();
    // Tell the user, then create a new call.
  }
};

A call also ends when the browser's connection is lost for more than a few seconds, and when the media never connects. KugelAudio updates the service without ending the calls that are running: a server that is being replaced takes no new calls and keeps its running ones until they end.

Errors when creating a call

The create call answers with a JSON body, {"error": {"code", "message", "type"}}.

StatusCodeMeaning
400invalid_json, invalid_transport, invalid_sdp, invalid_sessionThe body is not a create request. The message says which part.
400unsupported_on_webrtcsession has a field that WebRTC does not take: see what differs.
401invalid_api_keyThe key is missing or was refused: it is invalid, lacks the Realtime sessions permission, its organization is not enabled, or the credits are spent.
413body_too_largeThe body is too large.
502upstream_unavailableThe session could not be opened. Retry.
503server_busy, server_drainingThis server takes no new calls right now. Retry: another server answers.

Retry a 502 or a 503 up to three times with a short pause. The other fields of session are checked when the call connects. A refusal then arrives as an error event on the data channel, and the call ends: an unknown model, for example, or a rate limit.

Networks and firewalls

The audio uses UDP port 3478. On a network that blocks UDP, the browser falls back to TCP port 443 by itself. A company firewall has to allow one of the two for outgoing connections.

Next steps