Skip to content
KugelAudioDocs

Streaming

Streamed audio for one request, streaming sessions for LLM output, barge-in, multi-context sessions, and word timestamps

The SDKs stream in two ways:

  • One request, streamed output: send the whole text and receive audio chunks as they are generated.
  • Streaming sessions for real-time LLM pipelines: a persistent WebSocket on the session endpoint (/ws/tts/stream) that accepts text as the LLM produces it, accumulates the tokens server-side and starts generation at natural sentence boundaries.

Which to choose, and the wire format behind both, is in the Streaming guide. In the examples on this page, play_audio, playAudio, stop_local_playback and stopLocalPlayback stand for your own playback functions; the SDKs do not ship them.

Stream one request

Receive audio chunks as they are generated for lower latency:

from kugelaudio import AudioChunk

# Synchronous streaming
for item in client.tts.stream(
    text="Hello, this is streaming audio.",
    model_id="kugel-3.5",
    voice_id=1071,
):
    if isinstance(item, AudioChunk):
        # Process audio chunk immediately
        print(f"Chunk {item.index}: {len(item.audio)} bytes, {item.samples} samples")
        play_audio(item.audio)
    elif isinstance(item, dict) and item.get('final'):
        # Final stats
        print(f"Total duration: {item.get('dur_ms', 0):.0f}ms")
        print(f"Generation time: {item.get('gen_ms', 0):.0f}ms")

For async applications, stream_async() yields the same items:

Python
import asyncio

from kugelaudio import AudioChunk

async def generate_speech():
    async for item in client.tts.stream_async(
        text="Async streaming example.",
        model_id="kugel-3.5",
        voice_id=1071,
    ):
        if isinstance(item, AudioChunk):
            play_audio(item.audio)

asyncio.run(generate_speech())

stream_async() reuses the client's pooled WebSocket by default; pass reuse_connection=False to open a fresh one.

Pipe into a Node Readable

Not available in the Python SDK; iterate stream() or stream_async() and write the chunks to your response.

Play chunks in the browser

Not applicable: the Python SDK runs on your server.

Streaming sessions for LLM output

For real-time TTS while an LLM streams its answer, open a streaming session and send the tokens as they arrive:

import asyncio

async def stream_from_llm():
    # Simulate LLM token stream
    llm_tokens = ["Hello, ", "this ", "is ", "a ", "streamed ", "response."]

    async with client.tts.streaming_session(
        voice_id=1071,
        cfg_scale=2.0,
        flush_timeout_ms=500,  # Server-side auto-flush timeout
    ) as session:
        # Send tokens as they arrive from LLM
        for token in llm_tokens:
            async for chunk in session.send(token):
                # Play audio chunk immediately
                play_audio(chunk.audio)

        # Flush any remaining text
        async for chunk in session.flush():
            play_audio(chunk.audio)

asyncio.run(stream_from_llm())

A synchronous session has the same shape without await:

Python
with client.tts.streaming_session_sync(voice_id=1071) as session:
    for token in llm_tokens:
        for chunk in session.send(token):
            play_audio(chunk.audio)

    for chunk in session.flush():
        play_audio(chunk.audio)

Session reuse

End a session without closing the WebSocket to avoid reconnection overhead when starting a new one (see Turn lifecycle):

async with client.tts.streaming_session(voice_id=1071) as session:
    # Session 1
    async for chunk in session.send("Hello from voice one.", flush=True):
        play_audio(chunk.audio)
    async for chunk in session.drain():  # receive the tail audio
        play_audio(chunk.audio)
    await session.end_session()  # Keeps WebSocket open

    # Session 2: no reconnection needed
    session.update_config(voice_id=1072)
    async for chunk in session.send("Hello from voice two.", flush=True):
        play_audio(chunk.audio)
    async for chunk in session.drain():
        play_audio(chunk.audio)
# Leaving the block closes the session and the WebSocket

end_session() and close() discard audio that is still in flight, so call drain() first to receive the rest of the turn. Session reuse needs the async session: StreamingSessionSync has no end_session() or update_config().

Barge-in (interrupt the current turn)

When the end user speaks over the agent, cancel the current turn: generation stops immediately and any buffered or queued text is dropped, without closing the WebSocket. Unlike ending the session, no remaining text is synthesized; the turn is abandoned. The socket stays open so the next send() starts the next turn right away.

session = client.tts.streaming_session(voice_id=1071)

async for chunk in session.send("This is a very long answer the user talks over"):
    play_audio(chunk.audio)

# VAD detected the user speaking: barge in
await session.cancel_current()

# Socket still open: next turn starts immediately
async for chunk in session.send("Sure, what would you like instead?", flush=True):
    play_audio(chunk.audio)

Python's cancel_current() and TypeScript's cancelCurrent() return once the server acknowledges (TypeScript's onInterrupted fires), or after a short quiet timeout if the server goes silent. Java's cancelCurrent() blocks until the acknowledgement (onInterrupted() fires), or up to about 5 seconds. Rust's cancel() and Dart's cancel() return once the server acknowledges, or after 30 seconds of silence; late audio of the cancelled turn is discarded. In Rust, drop the audio stream you are reading before you call cancel(), since both borrow the session. C++'s cancel() returns once the server acknowledges and is the one session method you may call from another thread while send() or flush() is blocked. Stop local playback as soon as you call it: a few in-flight frames may arrive before the acknowledgement. Python's synchronous session exposes cancel_current() too. See Barge-in for the full protocol.

Updating settings mid-session

Change generation parameters on a live connection without reconnecting. The update is explicit and acknowledged, and returns the parameters now in effect:

session = client.tts.streaming_session(voice_id=1071, cfg_scale=2.0)

async for chunk in session.send("Default voicing.", flush=True):
    play_audio(chunk.audio)

# Change cfg_scale / speed / temperature for the next turn:
effective = await session.update_settings(cfg_scale=1.5, speed=1.1)
# -> {"cfg_scale": 1.5, "temperature": ..., "speed": 1.1, ...}

async for chunk in session.send("Now a bit faster and softer.", flush=True):
    play_audio(chunk.audio)

Only generation parameters are updatable: CFG scale, temperature, speed, max new tokens, language and normalize. Identity and audio-format settings (voice, model, sample rate, output format, project, dictionary IDs) are fixed for the connection. The change applies to the next turn; call it between turns. Multi-context sessions expose the same update, session-scoped: it applies to contexts started after the update.

update_settings(**params) takes cfg_scale, temperature, speed, max_new_tokens, language, normalize and returns a dict. Change identity settings with update_config() after end_session(). The server rejects an out-of-range value or a non-updatable field with a typed error. StreamingSessionSync exposes update_settings() too.

Server-side chunking

The server accumulates tokens and synthesizes each sentence as soon as it ends. Two session settings tune its buffer; what they do is in Chunking and latency. The auto-flush timeout defaults to 500 ms and the maximum buffer to 1000 characters:

SDKAuto-flush timeout (ms)Maximum buffered characters
Pythonflush_timeout_ms (session factory argument)max_buffer_length (set with session.update_config(...) before the first send)
TypeScriptflushTimeoutMsmaxBufferLength
Java.flushTimeoutMs(int)not available
RustStreamConfig::flush_timeout_ms(u32)StreamConfig::max_buffer_length(u32)
C++StreamConfig::flush_timeout_msStreamConfig::max_buffer_length
DartStreamConfig(flushTimeoutMs: ...)StreamConfig(maxBufferLength: ...)

The SDKs also accept the ElevenLabs-style chunk_length_schedule / auto_mode (Python, and the Rust and C++ StreamConfig), chunkLengthSchedule / autoMode (TypeScript and Dart) and .chunkLengthSchedule(...) / .autoMode(...) (Java) for compatibility. They have no effect: the streaming socket ignores both and answers with an "Unknown fields ignored" warning.

Session reference

A session is created with streaming_session(...) (async) or streaming_session_sync(...) (sync). Both accept the same configuration: voice_id, model_id, cfg_scale, temperature, max_new_tokens, sample_rate, flush_timeout_ms, normalize, language, word_timestamps, speed, dictionary_ids, project_id, and an on_word_timestamps callback. Although voice_id defaults to None in the SDK signature, it must be set before the first synthesis; omission fails with MISSING_VOICE_ID. project_id and dictionary_ids select dictionaries (see Dictionaries at synthesis). The full field list is StreamConfig.

The async StreamingSession exposes:

MethodReturnsDescription
await session.connect()NoneOpen and authenticate the WebSocket. Called automatically by the first send() and by async with.
session.send(text, flush=False)AsyncIterator[AudioChunk]Buffer text and yield audio as it is generated. Without flush, it yields only what arrives within a short poll; flush=True forces synthesis of whatever is buffered and waits for its audio.
session.flush()AsyncIterator[AudioChunk]Flush the buffer and yield remaining audio for the current turn.
session.drain()AsyncIterator[AudioChunk]Signal end-of-input and yield every remaining chunk until the server closes the session. Call it before end_session() or close() to receive the tail audio.
await session.end_session()dictEnd the current session but keep the WebSocket open for reuse. The server flushes buffered text, but audio still in flight is discarded client-side; call drain() first to receive it.
await session.cancel_current()NoneBarge-in: abandon the current turn and drop buffered or queued text, keeping the socket open.
session.update_config(config=None, **kwargs)NoneUpdate configuration (e.g. voice_id) for the next session after end_session(). Keyword arguments change only those fields; passing a whole StreamConfig replaces every field.
await session.update_settings(**params)dictChange generation parameters (cfg_scale, temperature, speed, max_new_tokens, language, normalize) mid-connection and return the effective values. Applies to the next turn.
await session.close()dictCancel a pending connection attempt, or end the session and close the WebSocket. Like end_session(), it discards audio still in flight.
session.last_word_timestampslist[WordTimestamp]The most recently received word timestamps.
session.last_finaldict | NoneEnd-of-audio stats from the most recently completed turn: the server's {"final": true, ...} frame (ElevenLabs isFinal equivalent), sent after the turn's last audio frame. None before the first turn completes; not updated on a barge-in cancel.
session.last_usageSessionUsage | NonePer-session usage (audio time + amount charged) from the most recently closed session, for billing your own customers per conversation. None before the first session closes. See SessionUsage.

StreamingSessionSync has a subset of this API without await/async for: send(), flush(), and drain() return list[AudioChunk], and cancel_current(), update_settings(), close(), and the last_word_timestamps / last_final / last_usage properties behave the same. It has no connect(), end_session(), or update_config(), so session reuse and voice switching are async-only.

Per-session usage

For billing your own customers per conversation, every closed session reports a SessionUsage: the audio time and the actual amount charged. One-shot requests and multi-context contexts report it too.

async with client.tts.streaming_session(voice_id=1071) as session:
    async for chunk in session.send("Hello!", flush=True):
        play_audio(chunk.audio)

usage = session.last_usage  # None before the first session closes
if usage is not None:
    print(f"audio: {usage.audio_seconds:.1f}s")
    if usage.cost_available:
        print(f"charged: {usage.cost_cents:.2f} {usage.currency} ct")
SDKStreaming session (most recently closed)One-shot requestMulti-context context
Pythonsession.last_usageAudioResponse.usage (generate())session.usage_for(context_id)
TypeScriptsession.lastUsageGenerationStats.usage in onFinal (stream())session.usageFor(contextId), or the onContextClosed argument
Javasession.getLastUsage()response.getUsage() on the AudioResponse of generate() / stream()session.getUsageFor(contextId)
Rustsession.last_usage()FinalStats.usage of StreamEvent::Final, or AudioResponse.usage from collect_response() (stream())session.usage_for(context_id)
C++session.last_usage()AudioResponse::usage of stream()session.usage_for(context_id)
Dartsession.lastUsage, or TurnResult.usage from flush()(await stream.result).usage, or AudioResponse.usage from collect() (stream())session.usageFor(contextId), or the result of closeContext()

Multi-context sessions

A multi-context session manages up to 20 independent audio-generation contexts over a single WebSocket (see limits). Each context has its own text buffer, voice, and generation queue: useful for multi-speaker conversations, pre-buffering one stream while another plays, or interleaving audio for dynamic dialogue. The concepts are in the Multi-context guide.

async with client.tts.multi_context_session(language="en") as session:
    # Create contexts, optionally with different voices
    await session.create_context("narrator", voice_id=1071)
    await session.create_context("character", voice_id=1072)

    # Send text to a specific context
    async for chunk in session.send("narrator", "The story begins."):
        play_audio(chunk.audio)

    async for chunk in session.send("character", "Hello there!", flush=True):
        play_audio(chunk.audio)

    # Drain remaining audio and close one context
    async for chunk in session.close_context("narrator"):
        play_audio(chunk.audio)

Every context that synthesizes text must have an effective voice; a context without one fails on its first text with MISSING_VOICE_ID. To barge in on one speaker (the user talks over speaker A), close only that context immediately: its in-flight generation and queued text are dropped, and the other contexts and the connection stay open.

async for chunk in session.close_context("narrator", immediate=True):
    play_audio(chunk.audio)

Multi-context reference

Create the session with multi_context_session(...):

ParameterTypeDefaultDescription
default_voice_idint | NoneNoneDefault voice for contexts that don't override it.
model_idstr | NoneNoneModel to use.
sample_rateint24000Output sample rate.
cfg_scalefloat2.0Guidance scale (1.2 to 2.5; values outside are clamped).
temperaturefloat | NoneNoneSampling variance.
max_new_tokensint2048Maximum tokens per generation.
normalizeboolTrueEnable text normalization.
languagestr | NoneNoneNormalization language.
dictionary_idslist[int] | NoneNoneExact dictionary selection; requires project_id. [] disables dictionaries.
inactivity_timeoutfloat20.0Seconds before an idle context auto-closes.
word_timestampsboolFalseRequest per-word time alignments for every context.
on_word_timestampsCallable[[str, list[WordTimestamp]], None] | NoneNoneCalled with (context_id, timestamps) as alignments arrive.
project_idint | NoneNoneProject whose dictionaries apply; see Dictionaries at synthesis.

Set default_voice_id on the session or pass voice_id to every create_context() call.

MultiContextSession methods:

MethodReturnsDescription
await session.connect()NoneOpen the WebSocket. Called automatically by async with.
await session.create_context(context_id, voice_id=None)NoneCreate a context with an optional voice override.
session.send(context_id, text, flush=False, chunk_complete_idle_timeout=None)AsyncIterator[AudioChunk]Send text to a context and yield its audio.
session.flush(context_id)AsyncIterator[AudioChunk]Flush a context's buffer.
session.close_context(context_id, immediate=False)AsyncIterator[AudioChunk]Close a context and drain its audio. immediate=True barges in, discarding buffered or queued text.
await session.keep_alive(context_id)NoneReset a context's inactivity timeout.
await session.update_settings(**params)dictChange generation parameters for contexts started after the update; returns the effective values.
session.get_word_timestamps(context_id)list[WordTimestamp]The most recently received word timestamps for a context.
await session.close()dictCancel a pending connection attempt, or close the established session and return stats.
session.usage_for(context_id)SessionUsage | NonePer-context usage (audio time + amount charged) for a closed context; each context is its own conversation. None until that context closes. See SessionUsage.
session.context_usagedict[str, SessionUsage]Map of context_id to usage for every context closed so far.
session.active_contextsset[str]The set of currently active context IDs.
session.session_idstr | NoneServer-assigned session ID.
session.is_aliveboolWhether the underlying WebSocket is still usable for send().

Word timestamps in streaming

Word timestamps work with one-shot streams and streaming sessions. They add no extra audio latency: they arrive shortly after the corresponding audio chunk (see Latency) and are useful for barge-in handling, subtitle synchronization, and lip-sync.

One-shot stream:

from kugelaudio import AudioChunk, WordTimestamp

# Timestamps are yielded as list[WordTimestamp] items between audio chunks
for item in client.tts.stream(
    text="Hello, how are you today?",
    model_id="kugel-3.5",
    voice_id=1071,
    word_timestamps=True,
):
    if isinstance(item, AudioChunk):
        play_audio(item.audio)
    elif isinstance(item, list) and item and isinstance(item[0], WordTimestamp):
        for ts in item:
            print(f"{ts.word}: {ts.start_ms}-{ts.end_ms}ms")

Streaming session: timestamps are delivered per chunk after the corresponding audio data.

async with client.tts.streaming_session(
    voice_id=1071,
    word_timestamps=True,
) as session:
    async for chunk in session.send("Hello, how are you today?"):
        play_audio(chunk.audio)

    async for chunk in session.flush():
        play_audio(chunk.audio)

    # Access the latest word timestamps
    timestamps = session.last_word_timestamps
    for ts in timestamps:
        print(f"{ts.word}: {ts.start_ms}ms - {ts.end_ms}ms (score: {ts.score:.2f})")

# Or register a callback to process timestamps as they arrive
def on_timestamps(timestamps):
    for ts in timestamps:
        print(f"  {ts.word} [{ts.start_ms}-{ts.end_ms}ms]")

async with client.tts.streaming_session(
    voice_id=1071,
    on_word_timestamps=on_timestamps,
) as session:
    async for chunk in session.send("Hello world!"):
        play_audio(chunk.audio)
    async for chunk in session.flush():
        play_audio(chunk.audio)

Multi-context sessions deliver timestamps per context: on_word_timestamps and get_word_timestamps(context_id) in Python, onWordTimestamps(contextId, timestamps) in Java, on_word_timestamps and word_timestamps(context_id) in Rust, MultiContextCallbacks::on_word_timestamps and word_timestamps(context_id) in C++, and the wordTimestamps stream and wordTimestampsFor(contextId) in Dart. The TypeScript multi-context session has no word-timestamp option.

Reconnecting after a server restart

During a rolling deploy the server closes sockets with code 1012 or 1013 (see Errors for the close codes).

Sessions retry silently when no audio has been delivered yet. When the session cannot replay the turn without repeating audio, it raises ServerRestartingError (a ConnectionError subclass): reconnect and resend the in-flight turn after retry_after seconds. See Errors.

Next steps