Skip to content
KugelAudioDocs

Follow and steer a call from your server

Attach your server to a running session over a second WebSocket: follow its events, answer delegations and change the instructions while the audio travels elsewhere.

On this page

A sideband is a second WebSocket on a session that is already running. Use it when the audio does not pass through your server, as on a WebRTC call: the browser holds the call, and your server follows it and tells the agent what it needs to know.

It follows the sideband of OpenAI's server controls.

Browser
Holds the call
WebRTC: audio and events
Fluid 1
sideband: events and commands
Your server
Holds the key
The audio goes direct; your server attaches to the same session

Attach to a session

HTTP
GET wss://api.eu.kugelaudio.com/v1/live/sessions/SESSION_ID/attach
Authorization: Bearer YOUR_API_KEY

SESSION_ID is the session.id you got when you created the WebRTC call. A session on a WebSocket has one too, in session.started.

  • Use a key of the project that started the session. To any other key the session does not exist.
  • Attach on the host that started the session: api.eu.kugelaudio.com for a WebRTC call, as in the examples here.
  • A session takes one sideband at a time. After yours closes, you can attach again.

You can attach as soon as the call is created, before the browser has connected. Your server then sees the session from its first event, session.started.

Follow the call

Every frame is a JSON text frame. You get each server event of the session except session.output_audio.delta: the transcripts of both sides, the delegations, usage and session.closed. The first frames are the events of the last three seconds before you attached, so a sideband that attaches a moment late misses nothing.

This server answers every delegation of one call. It uses the websockets package:

sideband.py
import asyncio
import json
import os
import sys

from websockets.asyncio.client import connect


async def follow(session_id: str) -> None:
    async with connect(
        f"wss://api.eu.kugelaudio.com/v1/live/sessions/{session_id}/attach",
        additional_headers={"Authorization": f"Bearer {os.environ['KUGELAUDIO_API_KEY']}"},
    ) as sideband:
        caller_said = ""
        async for frame in sideband:
            event = json.loads(frame)
            if event["type"] == "session.input_transcript.delta":
                caller_said += event["delta"]
            elif event["type"] == "session.delegation.created":
                print("The caller said:", caller_said)
                await sideband.send(json.dumps({
                    "type": "session.commentary.append",
                    "delegation_id": event["delegation"]["id"],
                    "content": "Order 4711 shipped yesterday and arrives tomorrow.",
                }))
                caller_said = ""
            elif event["type"] == "session.closed":
                break


asyncio.run(follow(sys.argv[1]))

A delegation is created only on a session that asked for it: put "delegation": {"type": "client"} into session when you create the call. What to send back is on Answer from your backend.

Steer the call

Send any client event except session.start. It acts on the session as if the call's own connection had sent it.

ToSend
Answer a delegationsession.commentary.append with its delegation_id
Give the agent silent contextsession.thinking.append
Change how the agent behavessession.instructions.append, or session.update with new instructions
Have the agent say a linesession.commentary.append with "delegation_id": null
Stop listening for a whilesession.input_audio.mute, then session.input_audio.unmute
End the callsession.close
Python
await sideband.send(json.dumps({
    "type": "session.instructions.append",
    "delegation_id": None,
    "content": "The caller's identity is confirmed. You can now give delivery details.",
}))

The session is already started, so a session.start on the sideband is answered with an error event with the code session_already_started, and the session goes on.

The browser's data channel gets the same events and can send the same commands. The page is in the caller's hands, so keep in your server what decides the agent's answers.

When the sideband closes

Closing the sideband does not end the call. To end the call, send session.close first.

Close codeReasonMeaning
1000session closedThe session ended. session.closed came before it.
1013sideband fell behindYour server read the events too slowly. Attach again.
1013try againThe session could not be looked up right now. Attach again after a pause.
4404session not foundNo running session has this id for your project: it ended, the key belongs to another project, or the session runs on another host.
4409a sideband is already attachedAnother connection of yours is attached. Close it first.

A key that is missing, invalid or not allowed is refused before the upgrade with HTTP 403, as on the session WebSocket.

If your sideband drops during a call, attach again. The call went on without it. The last three seconds of events are sent again, so your server can see an event twice.

Next steps