> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kugelaudio.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Connect a browser with WebRTC

> Let a web page or an app talk to the agent over WebRTC: your server creates the call with your key, and the audio travels straight between the browser and KugelAudio.

With WebRTC the caller's browser sends its microphone straight to KugelAudio and plays the
agent's audio as it arrives. Your server carries no audio. It creates the call with your API
key.

Your server can also [follow and steer the call](/speech-to-speech/sideband) over a second
connection.

The call follows the WebRTC part of the
[OpenAI Live API](https://developers.openai.com/api/docs/guides/voice-webrtc?api=live). Use
the [WebSocket](/speech-to-speech/api-reference) when the audio already passes through your
server, as it does on a phone line.

## How a call is set up

<Flow caption="Your server creates the call; the audio does not pass through it">
  <FlowNode title="Browser" icon="globe">Microphone and speaker</FlowNode>
  <FlowLink label="1. offer, 4. answer" both />
  <FlowNode title="Your server" icon="server">Checks who the user is, holds the key</FlowNode>
  <FlowLink label="2. create the call, 3. answer" both />
  <FlowNode title="Fluid 1" icon="waveform-lines" highlight />
</Flow>

<Flow caption="Then the audio and the events go direct">
  <FlowNode title="Browser" icon="globe" />
  <FlowLink label="WebRTC: audio and events" both />
  <FlowNode title="Fluid 1" icon="waveform-lines" highlight />
</Flow>

1. The browser creates a WebRTC offer and sends it to your server.
2. Your server checks that this user may start a call and posts the offer, with the session
   settings and your API key, to `/v1/live/sessions`.
3. KugelAudio returns an answer and the session id.
4. Your server passes the answer to the browser. The browser connects, and the agent starts.

Your key never reaches the browser. Anyone who loads a page can read a key that is in it, so
the browser cannot create a call on its own.

## Create the call on your server

WebRTC calls run in the EU. Create them on `api.eu.kugelaudio.com`, wherever your server
runs: the US host does not take them.

```http
POST https://api.eu.kugelaudio.com/v1/live/sessions
Authorization: Bearer YOUR_API_KEY
Content-Type: application/json
```

```json
{
  "session": {
    "model": "fluid-1",
    "instructions": "You answer for a parcel service. Keep replies short."
  },
  "transport": { "type": "webrtc", "sdp": "v=0\r\n..." }
}
```

`session` takes the fields of [`session.start`](/speech-to-speech/api-reference#session-start),
except those listed under [what differs](#what-differs-from-the-websocket). `transport.sdp` is
the browser's offer, unchanged. The answer is `201`:

```json
{
  "session": { "id": "live_3f2a9c1d5b7e4a60812c9d4e6f7a8b90" },
  "transport": { "type": "webrtc", "sdp": "v=0\r\n..." }
}
```

Give `transport.sdp` to the browser. `session.id` names the call.

Keep `session.id` if your server will [attach to the call](/speech-to-speech/sideband).

A minimal endpoint, with FastAPI and `httpx`:

```python calls.py
import os

import httpx
from fastapi import FastAPI, HTTPException, Request, Response

app = FastAPI()


@app.post("/calls")
async def create_call(request: Request) -> Response:
    # Check here that this user may start a call.
    offer = (await request.body()).decode()
    async with httpx.AsyncClient(timeout=20) as client:
        created = await client.post(
            "https://api.eu.kugelaudio.com/v1/live/sessions",
            headers={"Authorization": f"Bearer {os.environ['KUGELAUDIO_API_KEY']}"},
            json={
                "session": {
                    "model": "fluid-1",
                    "instructions": "You answer for a parcel service. Keep replies short.",
                },
                "transport": {"type": "webrtc", "sdp": offer},
            },
        )
    if created.status_code != 201:
        raise HTTPException(status_code=502, detail=created.json()["error"]["code"])
    call = created.json()
    # call["session"]["id"] names the call.
    return Response(call["transport"]["sdp"], media_type="application/sdp")
```

## Connect from the browser

```javascript call.js
const connection = new RTCPeerConnection();

// Play the agent.
const speaker = new Audio();
speaker.autoplay = true;
connection.ontrack = (event) => {
  speaker.srcObject = new MediaStream([event.track]);
};

// Send the microphone.
const microphone = await navigator.mediaDevices.getUserMedia({ audio: true });
connection.addTrack(microphone.getAudioTracks()[0], microphone);

// Events travel on a data channel with this name.
const events = connection.createDataChannel("oai-events");
events.onmessage = (message) => {
  const event = JSON.parse(message.data);
  console.log(event.type);
};

await connection.setLocalDescription(await connection.createOffer());
const created = await fetch("/calls", {
  method: "POST",
  headers: { "Content-Type": "application/sdp" },
  body: connection.localDescription.sdp,
});
await connection.setRemoteDescription({ type: "answer", sdp: await created.text() });
```

The browser needs no STUN or TURN server, so `RTCPeerConnection` takes no `iceServers`. Call
`getUserMedia` after a click or a tap: browsers ask for the microphone, and most refuse to play
audio on a page the user has not touched.

The session starts when the connection is up, so the agent's greeting is not spoken before
anyone can hear it. To hang up, call `connection.close()`.

## Events on the data channel

Every [server event](/speech-to-speech/api-reference#server-events) except the audio arrives
on `oai-events` as JSON text, and the page can send
[client events](/speech-to-speech/api-reference#client-events) the same way:

```javascript
events.send(JSON.stringify({ type: "session.input_audio.mute" }));
```

The page is in the caller's hands, so do not let it decide what the agent is told.

Answer [delegations](/speech-to-speech/your-backend) and change the instructions from your
server, over the [sideband](/speech-to-speech/sideband).

## What differs from the WebSocket

| Topic | Over WebRTC |
|---|---|
| Starting the session | The create call starts it. A `session.start` on the data channel is answered with an `error` event. |
| Audio | Opus on the media tracks, both ways. `session.input_audio.append` is answered with an `error` event, and no `session.output_audio.delta` arrives. |
| `session.audio.format` | Set by the transport. Sending it is refused with `unsupported_on_webrtc`. |
| `session.kugel` and the `kugel.*` events | Not available. Sending `session.kugel` is refused with `unsupported_on_webrtc`, so the call starts in the language of your instructions. |
| `session.client` | Not available. Refused with `unsupported_on_webrtc`. |
| Interruptions | Nothing to do. The audio arrives at the pace of speech, and the agent's speech stops in the stream when the caller cuts in. |
| Limits and usage | The [same](/speech-to-speech/limits) as a WebSocket session. A call is one session. |

## When a call ends or the connection breaks

A call lives on one connection. It cannot move to another network path or be resumed, so when
`connection.connectionState` becomes `failed` or `closed`, or `session.closed` arrives, the
call is over. To carry on, create a new call and pass what was said in
[`input`](/speech-to-speech/during-a-call#continue-after-a-session-ended).

```javascript
connection.onconnectionstatechange = () => {
  if (connection.connectionState === "failed") {
    connection.close();
    // Tell the user, then create a new call.
  }
};
```

A call also ends when the browser's connection is lost for more than a few seconds, and when
the media never connects. KugelAudio updates the service without ending the calls
that are running: a server that is being replaced takes no new calls and keeps its running
ones until they end.

## Errors when creating a call

The create call answers with a JSON body, `{"error": {"code", "message", "type"}}`.

| Status | Code | Meaning |
|---|---|---|
| `400` | `invalid_json`, `invalid_transport`, `invalid_sdp`, `invalid_session` | The body is not a create request. The message says which part. |
| `400` | `unsupported_on_webrtc` | `session` has a field that WebRTC does not take: see [what differs](#what-differs-from-the-websocket). |
| `401` | `invalid_api_key` | The key is missing or was refused: it is invalid, lacks the **Realtime sessions** [permission](/api-reference/authentication#key-permissions), its organization is not enabled, or the credits are spent. |
| `413` | `body_too_large` | The body is too large. |
| `502` | `upstream_unavailable` | The session could not be opened. Retry. |
| `503` | `server_busy`, `server_draining` | This server takes no new calls right now. Retry: another server answers. |

Retry a `502` or a `503` up to three times with a short pause. The other fields of `session` are
checked when the call connects. A refusal then arrives as an
[`error` event](/speech-to-speech/api-reference#errors) on the data channel, and the call ends:
an unknown model, for example, or a [rate limit](/speech-to-speech/limits#retry-a-refused-session).

## Networks and firewalls

The audio uses UDP port 3478. On a network that blocks UDP, the browser falls back to TCP
port 443 by itself. A company firewall has to allow one of the two for outgoing connections.

## Next steps

<CardGroup cols={2}>
  <Card title="Follow and steer the call" icon="tower-broadcast" href="/speech-to-speech/sideband">
    Attach your server to a running call
  </Card>
  <Card title="Answer from your backend" icon="server" href="/speech-to-speech/your-backend">
    Look things up and have the agent tell the caller
  </Card>
  <Card title="Limits and usage" icon="gauge" href="/speech-to-speech/limits">
    How long a call can run and what counts as usage
  </Card>
</CardGroup>
