Control a call

Stop playback when the caller interrupts

A session without session.kugel needs nothing for this. Its audio arrives at the pace of speech, as 20 ms frames with silence between replies, so you play each frame as it comes. When the caller interrupts, the agent's speech stops in the stream and silence follows.

A session with "kugel": {} receives the kugel.* events and its audio as fast as it is generated, ahead of playback. Queue it, and when the agent's reply is cancelled, drop the audio you have not played yet:

playback = bytearray()  # audio waiting for your speaker or phone line

async for event in connection:
    if event.type == "session.output_audio.delta":
        playback.extend(base64.b64decode(event.delta))
    elif event.type == "kugel.response.cancelled":
        playback.clear()

The reference lists every kugel.* event.

Choose the voice

Set audio.output.voice to the numeric id of a voice. An OpenAI voice name selects the default KugelAudio voice and sends an info event with the code kugel_voice_alias.

session = {
    "model": "fluid-1",
    "instructions": "You answer the phone for a parcel service. Keep replies short.",
    "audio": {"output": {"voice": "973"}},
}

Choose the language

A call starts in session.kugel.language, a locale such as de-DE or en-US. Without it, the call starts in the language of your instructions when they are long enough to tell. Once the caller speaks, the agent follows the caller.

session = {
    "model": "fluid-1",
    "instructions": "Du nimmst Anrufe für einen Paketdienst an. Antworte kurz.",
    "kugel": {"language": "de-DE"},
}

Connect a phone line

Phone systems send 8 kHz G.711. Ask for it in audio.format, and the audio goes both ways in that format, so you can pass frames through without converting them:

session = {
    "model": "fluid-1",
    "instructions": "You answer the phone for a parcel service. Keep replies short.",
    "audio": {"format": {"type": "audio/pcmu", "rate": 8000}},
}

Use audio/pcma for A-law. Each session.input_audio.append carries base64 of the raw G.711 bytes.

Mute the microphone

Send session.input_audio.mute to stop listening without ending the call, for example while the caller is on hold. The agent hears silence, and its own speech goes on. Send session.input_audio.unmute to listen again.

await connection.send({"type": "session.input_audio.mute"})
await connection.send({"type": "session.input_audio.unmute"})

Each is answered with session.input_audio.muted or session.input_audio.unmuted.

End a session

Call connection.session.close(). The agent finishes the reply it is giving, and you get session.closed with the seconds used. A session lasts at most one hour: session.started gives the end as expires_at, and at that point you get session.closed with the reason expired.

Continue after a session ended

A session lives only on its socket. Nothing is stored, so a closed or broken socket ends the conversation and cannot resume it. To carry on, start a new session and pass what you kept in input. The agent then skips its greeting:

session = {
    "model": "fluid-1",
    "instructions": "You answer the phone for a parcel service. Keep replies short.",
    "input": [
        {"role": "user", "content": [{"type": "input_text", "text": "Where is order 4711?"}]},
        {"role": "assistant", "content": [{"type": "output_text", "text": "It arrives tomorrow."}]},
    ],
}

Your backend can finish an operation after the call ended, but it cannot add the result to a closed session.

Next steps