Follow and steer a call from your server
Attach your server to a running session over a second WebSocket: follow its events, answer delegations and change the instructions while the audio travels elsewhere.
On this page
A sideband is a second WebSocket on a session that is already running. Use it when the audio does not pass through your server, as on a WebRTC call: the browser holds the call, and your server follows it and tells the agent what it needs to know.
It follows the sideband of OpenAI's server controls.
Attach to a session
GET wss://api.eu.kugelaudio.com/v1/live/sessions/SESSION_ID/attach
Authorization: Bearer YOUR_API_KEYSESSION_ID is the session.id you got when you
created the WebRTC call. A session on
a WebSocket has one too, in session.started.
- Use a key of the project that started the session. To any other key the session does not exist.
- Attach on the host that started the session:
api.eu.kugelaudio.comfor a WebRTC call, as in the examples here. - A session takes one sideband at a time. After yours closes, you can attach again.
You can attach as soon as the call is created, before the browser has connected. Your server
then sees the session from its first event, session.started.
Follow the call
Every frame is a JSON text frame. You get each
server event of the session except
session.output_audio.delta: the transcripts of both sides, the delegations, usage and
session.closed. The first frames are the events of the last three seconds before you
attached, so a sideband that attaches a moment late misses nothing.
This server answers every delegation of one call. It uses the websockets package:
import asyncio
import json
import os
import sys
from websockets.asyncio.client import connect
async def follow(session_id: str) -> None:
async with connect(
f"wss://api.eu.kugelaudio.com/v1/live/sessions/{session_id}/attach",
additional_headers={"Authorization": f"Bearer {os.environ['KUGELAUDIO_API_KEY']}"},
) as sideband:
caller_said = ""
async for frame in sideband:
event = json.loads(frame)
if event["type"] == "session.input_transcript.delta":
caller_said += event["delta"]
elif event["type"] == "session.delegation.created":
print("The caller said:", caller_said)
await sideband.send(json.dumps({
"type": "session.commentary.append",
"delegation_id": event["delegation"]["id"],
"content": "Order 4711 shipped yesterday and arrives tomorrow.",
}))
caller_said = ""
elif event["type"] == "session.closed":
break
asyncio.run(follow(sys.argv[1]))A delegation is created only on a session that asked for it: put
"delegation": {"type": "client"} into session when you create the call. What to send back
is on Answer from your backend.
Steer the call
Send any client event except session.start.
It acts on the session as if the call's own connection had sent it.
| To | Send |
|---|---|
| Answer a delegation | session.commentary.append with its delegation_id |
| Give the agent silent context | session.thinking.append |
| Change how the agent behaves | session.instructions.append, or session.update with new instructions |
| Have the agent say a line | session.commentary.append with "delegation_id": null |
| Stop listening for a while | session.input_audio.mute, then session.input_audio.unmute |
| End the call | session.close |
await sideband.send(json.dumps({
"type": "session.instructions.append",
"delegation_id": None,
"content": "The caller's identity is confirmed. You can now give delivery details.",
}))The session is already started, so a session.start on the sideband is answered with an
error event with the code session_already_started, and the session goes on.
The browser's data channel gets the same events and can send the same commands. The page is in the caller's hands, so keep in your server what decides the agent's answers.
When the sideband closes
Closing the sideband does not end the call. To end the call, send session.close first.
| Close code | Reason | Meaning |
|---|---|---|
1000 | session closed | The session ended. session.closed came before it. |
1013 | sideband fell behind | Your server read the events too slowly. Attach again. |
1013 | try again | The session could not be looked up right now. Attach again after a pause. |
4404 | session not found | No running session has this id for your project: it ended, the key belongs to another project, or the session runs on another host. |
4409 | a sideband is already attached | Another connection of yours is attached. Close it first. |
A key that is missing, invalid or not allowed is refused before the upgrade with HTTP 403,
as on the session WebSocket.
If your sideband drops during a call, attach again. The call went on without it. The last three seconds of events are sent again, so your server can see an event twice.