Skip to content
KugelAudioDocs

Multi-context streaming

Guide to multi-context streaming: run several independent text-to-speech audio streams over one WebSocket for multi-speaker dialogue and pre-buffering.

On this page

For advanced use cases like multi-speaker conversations or pre-buffering audio, use the multi-context WebSocket endpoint (/ws/tts/multi). This allows managing up to 20 independent audio streams over a single connection.

Use cases

  • Multi-speaker conversations: Generate audio for different speakers concurrently
  • Pre-buffering: Start generating the next response while the current one plays
  • Interleaved audio: Dynamically switch between speakers in real-time

Example

Every SDK wraps /ws/tts/multi in a multi-context session. The raw WebSocket version follows below.

import asyncio
from kugelaudio import KugelAudio

client = KugelAudio(api_key="YOUR_API_KEY")


async def multi_speaker_demo():
    async with client.tts.multi_context_session(
        model_id="kugel-3.5", language="en"
    ) as session:
        await session.create_context("narrator", voice_id=1071)
        await session.create_context("character", voice_id=1072)

        async for chunk in session.send("narrator", "The story begins.", flush=True):
            play_audio("narrator", chunk.audio)
        async for chunk in session.send(
            "character", "Hello, I'm the main character!", flush=True
        ):
            play_audio("character", chunk.audio)

        # Gracefully drain and close both contexts
        for context_id in ("narrator", "character"):
            async for chunk in session.close_context(context_id):
                play_audio(context_id, chunk.audio)
            print(f"[{context_id}] usage: {session.usage_for(context_id)}")


asyncio.run(multi_speaker_demo())

Raw WebSocket

import asyncio
import websockets
import json
import base64

async def multi_speaker_demo():
    async with websockets.connect(
        "wss://api.kugelaudio.com/ws/tts/multi?api_key=YOUR_API_KEY"
    ) as ws:
        # The first message for an ID creates its context.
        await ws.send(json.dumps({
            "text": "The story begins.",
            "context_id": "narrator",
            "voice_settings": {"voice_id": 1071},
            "flush": True,
        }))

        await ws.send(json.dumps({
            "text": "Hello, I'm the main character!",
            "context_id": "character",
            "voice_settings": {"voice_id": 1072},
            "flush": True,
        }))

        # Gracefully drain and close both contexts.
        await ws.send(json.dumps({"close_context": True, "context_id": "narrator"}))
        await ws.send(json.dumps({"close_context": True, "context_id": "character"}))

        # Receive audio from both contexts
        closed = set()
        async for message in ws:
            data = json.loads(message)

            if "audio" in data:
                context_id = data["context_id"]
                audio_bytes = base64.b64decode(data["audio"])
                play_audio(context_id, audio_bytes)

            if data.get("context_closed"):
                # Per-context usage for this conversation: audio time + charge
                print(f"[{data['context_id']}] usage: {data.get('usage')}")
                closed.add(data["context_id"])
                if len(closed) == 2:
                    await ws.send(json.dumps({"close_socket": True}))

            if data.get("session_closed"):
                break

asyncio.run(multi_speaker_demo())

Protocol summary

Each context message carries a context_id; the first message for a new ID creates the context and can include voice_settings. An empty-text message can therefore create or keep alive a context, but voice_id must be set no later than the first non-empty text message. Per context you can send text, flush, and close_context (with "immediate": true for barge-in); {"close_socket": true} ends everything. The server tags every context-specific response frame (context_created, generation_started, audio, chunk_complete, word_timestamps, final, context_closed, and per-context errors) with the originating context_id. The final session_closed frame is connection-wide and has no context_id; it carries total_audio_seconds across the connection. After each flush, a final frame (ElevenLabs is_final equivalent) signals that all audio for the flushed text has been delivered.

If a context ends without producing any audio for text it accepted, the server replaces that final with a per-context error frame (error_code: "INTERNAL_ERROR", code: 500) before context_closed, so a lost turn is never indistinguishable from a completed one. Treat it as a failed turn and retry the text on a new context.

The full message tables, every field of every client→server and server→client frame, live in the Text-to-Speech API reference.

Usage is billed per context: a synthesized, billable context's context_closed frame carries its usage block. A context closed before synthesis starts has no usage block. See Per-session usage.

Limits

  • Maximum 20 concurrent contexts per connection
  • Contexts auto-close after 20 seconds of inactivity
  • Send empty text {"text": "", "context_id": "..."} to reset the per-context inactivity timeout
  • Opening a context beyond the limit returns a per-context error (error_code: "TOO_MANY_CONTEXTS", code: 429) without closing the connection. Close an existing context, or wait for an idle one to be released, then retry.