Streaming
Streamed audio for one request, streaming sessions for LLM output, barge-in, multi-context sessions, and word timestamps
The SDKs stream in two ways:
- One request, streamed output: send the whole text and receive audio chunks as they are generated.
- Streaming sessions for real-time LLM pipelines: a persistent WebSocket on the
session endpoint (
/ws/tts/stream) that accepts text as the LLM produces it, accumulates the tokens server-side and starts generation at natural sentence boundaries.
Which to choose, and the wire format behind both, is in the
Streaming guide. In the examples on this page, play_audio,
playAudio, stop_local_playback and stopLocalPlayback stand for your own
playback functions; the SDKs do not ship them.
Stream one request
Receive audio chunks as they are generated for lower latency:
from kugelaudio import AudioChunk
# Synchronous streaming
for item in client.tts.stream(
text="Hello, this is streaming audio.",
model_id="kugel-3.5",
voice_id=1071,
):
if isinstance(item, AudioChunk):
# Process audio chunk immediately
print(f"Chunk {item.index}: {len(item.audio)} bytes, {item.samples} samples")
play_audio(item.audio)
elif isinstance(item, dict) and item.get('final'):
# Final stats
print(f"Total duration: {item.get('dur_ms', 0):.0f}ms")
print(f"Generation time: {item.get('gen_ms', 0):.0f}ms")await client.tts.stream(
{
text: 'Hello, this is streaming audio.',
modelId: 'kugel-3.5',
voiceId: 1071,
},
{
onOpen: () => {
console.log('WebSocket connected');
},
onChunk: (chunk) => {
console.log(`Chunk ${chunk.index}: ${chunk.samples} samples`);
// chunk.audio uses chunk.encoding ('pcm_s16le', 'mulaw', or 'alaw')
playAudioChunk(chunk);
},
onFinal: (stats) => {
console.log(`Total duration: ${stats.durationMs}ms`);
console.log(`Generation time: ${stats.generationMs}ms`);
console.log(`RTF: ${stats.rtf}`);
},
onError: (error) => {
console.error('TTS error:', error);
},
onClose: () => {
console.log('WebSocket closed');
},
}
);import com.kugelaudio.sdk.GenerateRequest;
import com.kugelaudio.sdk.StreamCallbacks;
import com.kugelaudio.sdk.AudioChunk;
client.tts().stream(
GenerateRequest.builder("Hello, this is streaming audio.")
.modelId("kugel-3.5")
.voiceId(1071)
.language("en")
.build(),
new StreamCallbacks() {
@Override
public void onChunk(AudioChunk chunk) {
System.out.printf("Chunk %d: %d bytes, %d samples%n",
chunk.getIndex(), chunk.getAudio().length, chunk.getSamples());
// playAudio(chunk.getAudio());
}
@Override
public void onComplete(AudioResponse response) {
System.out.printf("Total duration: %.0fms%n", response.getDurationMs());
System.out.printf("Generation time: %.0fms%n", response.getGenerationMs());
}
@Override
public void onError(com.kugelaudio.sdk.KugelAudioException error) {
System.err.println("TTS error: " + error.getMessage());
}
}
);use futures_util::StreamExt;
use kugelaudio::{StreamEvent, TtsRequest};
let request = TtsRequest::new("Hello, this is streaming audio.")
.model_id("kugel-3.5")
.voice_id(1071)
.language("en");
let mut stream = client.tts().stream(&request).await?;
while let Some(event) = stream.next().await {
match event? {
StreamEvent::Audio(chunk) => {
// Process audio chunk immediately
println!("Chunk {}: {} bytes, {} samples", chunk.index, chunk.audio.len(), chunk.samples);
play_audio(&chunk.audio);
}
StreamEvent::Final(stats) => {
// Final stats
println!("Total duration: {:?}ms", stats.duration_ms);
println!("Generation time: {:?}ms", stats.generation_ms);
}
_ => {}
}
}kugelaudio::GenerateRequest request;
request.text = "Hello, this is streaming audio.";
request.model_id = "kugel-3.5";
request.voice_id = 1071;
request.language = "en";
kugelaudio::StreamCallbacks callbacks;
callbacks.on_audio = [](const kugelaudio::AudioChunk& chunk) {
// Process audio chunk immediately
std::cout << "Chunk " << chunk.index << ": " << chunk.audio.size() << " bytes, "
<< chunk.samples << " samples\n";
play_audio(chunk.audio);
};
// Blocks until the request is done; returns the collected audio and final stats
kugelaudio::AudioResponse result = client.tts().stream(request, callbacks);
std::cout << "Total duration: " << result.duration_ms << "ms\n";
std::cout << "Generation time: " << result.generation_ms << "ms\n";final stream = client.tts.stream(
text: 'Hello, this is streaming audio.',
modelId: 'kugel-3.5',
voiceId: 1071,
language: 'en',
);
await for (final chunk in stream) {
// Process audio chunk immediately
print('Chunk ${chunk.index}: ${chunk.audio.length} bytes, ${chunk.samples} samples');
playAudio(chunk.audio);
}
// Final stats
final stats = await stream.result;
print('Total duration: ${stats.durationMs}ms');
print('Generation time: ${stats.generationMs}ms');For async applications, stream_async() yields the same items:
import asyncio
from kugelaudio import AudioChunk
async def generate_speech():
async for item in client.tts.stream_async(
text="Async streaming example.",
model_id="kugel-3.5",
voice_id=1071,
):
if isinstance(item, AudioChunk):
play_audio(item.audio)
asyncio.run(generate_speech())stream_async() reuses the client's pooled WebSocket by default; pass
reuse_connection=False to open a fresh one.
stream() reuses the client's pooled WebSocket. Pass false as a third
argument (client.tts.stream(options, callbacks, false)) to open a fresh
connection for that call. The StreamCallbacks are all
optional.
tts().stream(request, callbacks, reuseConnection) takes an optional third
argument: pass true to reuse the client's pooled WebSocket connection instead of
opening a fresh one for the request. stream() blocks until the request completes.
The StreamCallbacks are shared with streaming sessions.
stream() returns a TtsStream, a futures::Stream of StreamEvents: Audio,
WordTimestamps and Final. It reuses the client's pooled WebSocket, or opens one
of its own when another request holds it, so requests may run concurrently.
Dropping the stream before Final closes its socket, so no late audio reaches a
later request. collect_response() gathers the whole stream into one
AudioResponse.
stream() blocks until the request completes and returns the collected
AudioResponse (set StreamOptions::collect_audio = false to skip gathering the
audio). Streams on the pooled socket run one at a time; set
StreamOptions::reuse_connection = false to open a connection of its own for
parallel streams. Call client.tts().cancel() from another thread to barge in:
stream() then returns with result.interrupted == true.
stream() returns a TtsStream, a Stream<AudioChunk> whose result future
carries the final stats and usage. Nothing is sent until you listen. It reuses the
client's pooled WebSocket; pass reuseConnection: false to open a fresh one.
Cancelling the subscription is a barge-in: the SDK tells the server to stop, drops
the late audio and keeps the socket. collect() gathers the stream into one
AudioResponse.
Pipe into a Node Readable
Not available in the Python SDK; iterate stream() or stream_async() and write the
chunks to your response.
For server-side integrations that expect a Node.js Readable stream, such as Vapi
custom TTS endpoints or Express/Fastify handlers, use client.tts.toReadable()
instead of wiring onChunk manually. It returns a Readable that emits raw binary
chunks in the requested format.
toReadable() avoids a common race condition: the stream object is returned
before any audio arrives, so you can safely pipe() or attach listeners
immediately.
import express from 'express';
import { KugelAudio } from 'kugelaudio';
const app = express();
app.use(express.json());
// Pre-connect at startup for fastest TTFA
const client = await KugelAudio.create({ apiKey: process.env.KUGELAUDIO_API_KEY! });
// Vapi custom TTS endpoint
app.post('/synthesize', (req, res) => {
const { text, sampleRate } = req.body.message;
res.setHeader('Content-Type', 'audio/pcm');
res.setHeader('Transfer-Encoding', 'chunked');
const readable = client.tts.toReadable({
text,
modelId: 'kugel-3.5',
voiceId: 1071,
sampleRate, // honour the sample rate Vapi requests
language: 'en',
});
readable.pipe(res); // pipe directly to an HTTP response or any writable
});Not available in the Java SDK; write the chunks from onChunk to your output stream.
Not available in the Rust SDK; TtsStream is already a futures::Stream, so map
its StreamEvent::Audio chunks into your HTTP framework's streaming body.
Not available in the C++ SDK; write the chunks from on_audio to your output
stream.
Not available in the Dart SDK; TtsStream is already a Stream<AudioChunk>, so
add its audio to any sink, for example
await response.addStream(stream.map((chunk) => chunk.audio)) on a dart:io
HttpResponse.
Play chunks in the browser
Not applicable: the Python SDK runs on your server.
import { decodePCM16, type AudioChunk } from 'kugelaudio';
const audioContext = new AudioContext();
// Use as the onChunk callback of stream() or a streaming session
function playPcmChunk(chunk: AudioChunk) {
// Convert base64 PCM16 to Float32 for the Web Audio API
const float32Data = decodePCM16(chunk.audio);
const audioBuffer = audioContext.createBuffer(1, float32Data.length, chunk.sampleRate);
audioBuffer.copyToChannel(float32Data, 0);
const source = audioContext.createBufferSource();
source.buffer = audioBuffer;
source.connect(audioContext.destination);
source.start();
}Not applicable: the Java SDK runs on your server.
Not applicable: the Rust SDK runs on your server or device.
Not applicable: the C++ SDK runs on your server or device.
The SDK leaves playback to you. Feed chunk.audio (mono PCM16 little-endian at
chunk.sampleRate) to a PCM streaming player package as it arrives, or use
chunk.toFloat32() for an audio graph. Never ship a project API key to a browser
(see Browser support).
Streaming sessions for LLM output
For real-time TTS while an LLM streams its answer, open a streaming session and send the tokens as they arrive:
import asyncio
async def stream_from_llm():
# Simulate LLM token stream
llm_tokens = ["Hello, ", "this ", "is ", "a ", "streamed ", "response."]
async with client.tts.streaming_session(
voice_id=1071,
cfg_scale=2.0,
flush_timeout_ms=500, # Server-side auto-flush timeout
) as session:
# Send tokens as they arrive from LLM
for token in llm_tokens:
async for chunk in session.send(token):
# Play audio chunk immediately
play_audio(chunk.audio)
# Flush any remaining text
async for chunk in session.flush():
play_audio(chunk.audio)
asyncio.run(stream_from_llm())const session = client.tts.streamingSession(
{
voiceId: 1071,
modelId: 'kugel-3.5',
},
{
onChunk: (chunk) => playAudio(chunk.audio),
onChunkComplete: (chunkId, audioSecs, genMs) => {
console.log(`Chunk ${chunkId}: ${audioSecs.toFixed(2)}s audio in ${genMs}ms`);
},
onSessionClosed: (totalSecs) => {
console.log(`Session complete: ${totalSecs.toFixed(2)}s total audio`);
},
onError: (err) => console.error('TTS error:', err),
}
);
await session.connect();
// Simulated LLM token stream: send tokens as they arrive
const llmTokens = ['Hello, ', 'this ', 'is ', 'a ', 'streamed ', 'response.'];
for (const token of llmTokens) {
session.send(token);
}
// Flush remaining buffer, deliver the remaining audio, and close
await session.close();import com.kugelaudio.sdk.StreamConfig;
import com.kugelaudio.sdk.StreamCallbacks;
import com.kugelaudio.sdk.StreamingSession;
import com.kugelaudio.sdk.AudioChunk;
StreamConfig config = StreamConfig.builder()
.voiceId(1071)
.modelId("kugel-3.5")
.language("en")
.flushTimeoutMs(500) // Server-side auto-flush timeout
.build();
// Simulate LLM token stream
String[] tokens = {"Hello, ", "this ", "is ", "a ", "streamed ", "response."};
try (StreamingSession session = client.streamingSession(config, new StreamCallbacks() {
@Override
public void onChunk(AudioChunk chunk) {
playAudio(chunk.getAudio());
}
@Override
public void onSessionClosed(double totalAudioSeconds, int totalTextChunks,
int totalAudioChunks) {
System.out.printf("Done: %.2fs%n", totalAudioSeconds);
}
})) {
// Send tokens as they arrive from LLM
for (String token : tokens) {
session.send(token);
}
// Leaving the block calls close(), which flushes the remaining text,
// waits for the audio, and fires onSessionClosed once.
}use futures_util::StreamExt;
use kugelaudio::StreamConfig;
// Simulate LLM token stream
let llm_tokens = ["Hello, ", "this ", "is ", "a ", "streamed ", "response."];
let mut session = client.tts().streaming_session(
StreamConfig::new(1071)
.model_id("kugel-3.5")
.language("en")
.flush_timeout_ms(500), // Server-side auto-flush timeout
);
session.connect().await?;
// Send tokens as they arrive from LLM
for token in llm_tokens {
let mut audio = session.send(token, false);
while let Some(chunk) = audio.next().await {
// Play audio chunk immediately
play_audio(&chunk?.audio);
}
}
// Flush the remaining text: ends the turn and yields its audio
let mut tail = session.flush();
while let Some(chunk) = tail.next().await {
play_audio(&chunk?.audio);
}
drop(tail);
session.close().await?;kugelaudio::StreamConfig config;
config.voice_id = 1071;
config.model_id = "kugel-3.5";
config.language = "en";
config.flush_timeout_ms = 500; // Server-side auto-flush timeout
auto play_chunk = [](const kugelaudio::AudioChunk& chunk) { play_audio(chunk.audio); };
// Simulate LLM token stream
const std::vector<std::string> llm_tokens = {"Hello, ", "this ", "is ", "a ", "streamed ", "response."};
kugelaudio::StreamingSession session = client.tts().streaming_session(config);
session.connect();
// Send tokens as they arrive from LLM; each call returns the audio ready so far
for (const std::string& token : llm_tokens) {
session.send(token, /*flush=*/false, play_chunk);
}
// Flush the remaining text: ends the turn and delivers its audio
session.flush(play_chunk);
session.close();final session = client.tts.streamingSession(
StreamConfig(
voiceId: 1071,
modelId: 'kugel-3.5',
language: 'en',
flushTimeoutMs: 500, // Server-side auto-flush timeout
),
);
// Audio of every turn arrives on one stream; listen before sending
session.audio.listen((chunk) => playAudio(chunk.audio));
await session.connect();
// Simulate LLM token stream: send tokens as they arrive
final llmTokens = ['Hello, ', 'this ', 'is ', 'a ', 'streamed ', 'response.'];
for (final token in llmTokens) {
await session.send(token);
}
// Flush the remaining text: completes once the turn's audio was delivered
await session.flush();
await session.close();A synchronous session has the same shape without await:
with client.tts.streaming_session_sync(voice_id=1071) as session:
for token in llm_tokens:
for chunk in session.send(token):
play_audio(chunk.audio)
for chunk in session.flush():
play_audio(chunk.audio)Audio arrives through the callbacks; send() returns immediately.
close() flushes the buffer and resolves once the remaining audio has been delivered.
Audio arrives through the callbacks; send() returns immediately. close() flushes
the remaining text, waits for its audio and fires onSessionClosed once.
send(), flush() and drain() return a stream of AudioChunks that borrows the
session; read it to the end (or drop it) before the next call. Without flush,
send() may end without audio after a short poll when the text has not completed a
sentence yet; keep sending. close() discards audio still in flight, so flush first.
send() and flush() call your callback with each chunk on the calling thread and
return once the audio ready so far has arrived; overloads without the callback
return a std::vector<AudioChunk>. Without flush, send() returns after a short
poll when the text has not completed a sentence yet. close() discards audio still
in flight, so flush first; the destructor closes the session too.
Audio of every turn arrives on session.audio, a single-subscription stream that
buffers until you listen. send() completes once the text is written. flush()
ends the turn and completes with its TurnResult once the audio was delivered.
speak(tokenStream) sends a whole token stream and flushes it.
Session reuse
End a session without closing the WebSocket to avoid reconnection overhead when starting a new one (see Turn lifecycle):
async with client.tts.streaming_session(voice_id=1071) as session:
# Session 1
async for chunk in session.send("Hello from voice one.", flush=True):
play_audio(chunk.audio)
async for chunk in session.drain(): # receive the tail audio
play_audio(chunk.audio)
await session.end_session() # Keeps WebSocket open
# Session 2: no reconnection needed
session.update_config(voice_id=1072)
async for chunk in session.send("Hello from voice two.", flush=True):
play_audio(chunk.audio)
async for chunk in session.drain():
play_audio(chunk.audio)
# Leaving the block closes the session and the WebSocketconst session = client.tts.streamingSession(
{ voiceId: 1071 },
{ onChunk: (chunk) => playAudio(chunk.audio) }
);
await session.connect();
// Session 1
session.send('Hello from voice one.');
await session.endSession(); // Delivers the remaining audio, keeps WebSocket open
// Session 2: no reconnection needed
session.updateConfig({ voiceId: 1072 });
session.send('Hello from voice two.');
await session.close(); // Closes session + WebSocketStreamingSession session = client.streamingSession(config, callbacks);
session.connect();
// Session 1
session.send("Hello from voice one.");
session.flush();
session.endSession(); // Keeps WebSocket open
// Session 2: no reconnection needed; the same StreamConfig is re-sent
session.send("Hello from the next turn.");
session.flush();
session.close(); // Closes session + WebSocketlet mut session = client.tts().streaming_session(StreamConfig::new(1071).language("en"));
session.connect().await?;
// Session 1
let mut audio = session.send("Hello from voice one.", true);
while let Some(chunk) = audio.next().await {
play_audio(&chunk?.audio);
}
drop(audio);
let mut tail = session.drain(); // receive the tail audio
while let Some(chunk) = tail.next().await {
play_audio(&chunk?.audio);
}
drop(tail);
session.end_session().await?; // Keeps WebSocket open
// Session 2: no reconnection needed
session.update_config(StreamConfig::new(1072).language("en"));
let mut audio = session.send("Hello from voice two.", true);
while let Some(chunk) = audio.next().await {
play_audio(&chunk?.audio);
}
drop(audio);
session.close().await?; // Closes session + WebSocketauto play_chunk = [](const kugelaudio::AudioChunk& chunk) { play_audio(chunk.audio); };
kugelaudio::StreamConfig config;
config.voice_id = 1071;
config.language = "en";
kugelaudio::StreamingSession session = client.tts().streaming_session(config);
session.connect();
// Session 1
session.send("Hello from voice one.", /*flush=*/true, play_chunk);
session.drain(play_chunk); // receive the tail audio; keeps WebSocket open
// Session 2: no reconnection needed
config.voice_id = 1072;
session.update_config(config);
session.send("Hello from voice two.", /*flush=*/true, play_chunk);
session.drain(play_chunk);
session.close(); // Closes session + WebSocketfinal session = client.tts.streamingSession(StreamConfig(voiceId: 1071, language: 'en'));
session.audio.listen((chunk) => playAudio(chunk.audio));
await session.connect();
// Session 1
await session.send('Hello from voice one.');
await session.flush(); // Delivers the remaining audio, keeps WebSocket open
// Session 2: no reconnection needed
session.updateConfig(StreamConfig(voiceId: 1072, language: 'en'));
await session.send('Hello from voice two.');
await session.flush();
await session.close(); // Closes session + WebSocketend_session() and close() discard audio that is still in flight, so call
drain() first to receive the rest of the turn. Session reuse needs the async
session: StreamingSessionSync has no end_session() or update_config().
endSession() resolves once the remaining audio has been delivered, so no drain
step is needed. updateConfig() changes the configuration (for example the voice)
for the next session.
The next session re-sends the same StreamConfig; Java has no updateConfig(). To
change the voice or another identity setting, create a new StreamingSession with
a fresh StreamConfig.
end_session() and close() discard audio that is still in flight, so call
drain() first to receive the rest of the turn. update_config() replaces the
whole StreamConfig for the next turn and keeps the socket.
drain() ends the turn, delivers its remaining audio and keeps the socket, so the
next send() starts a fresh turn. end_session() ends the turn too but discards
audio still in flight. update_config() replaces the whole StreamConfig for the
next turn.
flush() ends the turn and completes once its audio was delivered, so no drain step
is needed. updateConfig() replaces the whole StreamConfig for the next turn.
endSession() also ends the turn gracefully and keeps the socket.
Barge-in (interrupt the current turn)
When the end user speaks over the agent, cancel the current turn: generation stops
immediately and any buffered or queued text is dropped, without closing the
WebSocket. Unlike ending the session, no remaining text is synthesized; the turn is
abandoned. The socket stays open so the next send() starts the next turn right
away.
session = client.tts.streaming_session(voice_id=1071)
async for chunk in session.send("This is a very long answer the user talks over"):
play_audio(chunk.audio)
# VAD detected the user speaking: barge in
await session.cancel_current()
# Socket still open: next turn starts immediately
async for chunk in session.send("Sure, what would you like instead?", flush=True):
play_audio(chunk.audio)const session = client.tts.streamingSession(
{ voiceId: 1071 },
{
onChunk: (chunk) => playAudio(chunk.audio),
onInterrupted: () => stopLocalPlayback(),
}
);
await session.connect();
session.send('This is a very long answer the user talks over');
// VAD detected the user speaking: barge in
await session.cancelCurrent();
// Socket still open: next turn starts immediately
session.send('Sure, what would you like instead?', true);StreamCallbacks callbacks = new StreamCallbacks() {
@Override public void onChunk(AudioChunk chunk) { playAudio(chunk); }
@Override public void onInterrupted() { stopLocalPlayback(); }
};
StreamingSession session = client.streamingSession(config, callbacks);
session.connect();
session.send("This is a very long answer the user talks over");
// VAD detected the user speaking: barge in
session.cancelCurrent();
// Socket still open: next turn starts immediately
session.send("Sure, what would you like instead?", true);let mut session = client.tts().streaming_session(StreamConfig::new(1071).language("en"));
let mut audio = session.send("This is a very long answer the user talks over", false);
while let Some(chunk) = audio.next().await {
play_audio(&chunk?.audio);
}
drop(audio);
// VAD detected the user speaking: barge in
stop_local_playback();
session.cancel().await?;
// Socket still open: next turn starts immediately
let mut audio = session.send("Sure, what would you like instead?", true);
while let Some(chunk) = audio.next().await {
play_audio(&chunk?.audio);
}#include <thread>
auto play_chunk = [](const kugelaudio::AudioChunk& chunk) { play_audio(chunk.audio); };
kugelaudio::StreamingSession session = client.tts().streaming_session(config);
// The agent speaks on its own thread
std::thread speaker([&] {
session.send("This is a very long answer the user talks over", /*flush=*/true, play_chunk);
});
// VAD detected the user speaking: barge in from this thread
stop_local_playback();
session.cancel(); // the blocked send() returns early without the cancelled audio
speaker.join();
// Socket still open: next turn starts immediately
session.send("Sure, what would you like instead?", /*flush=*/true, play_chunk);final session = client.tts.streamingSession(StreamConfig(voiceId: 1071, language: 'en'));
session.audio.listen((chunk) => playAudio(chunk.audio));
await session.send('This is a very long answer the user talks over');
// VAD detected the user speaking: barge in
stopLocalPlayback();
await session.cancel(); // a pending flush() completes with null
// Socket still open: next turn starts immediately
await session.flush('Sure, what would you like instead?');Python's cancel_current() and TypeScript's cancelCurrent() return once the server
acknowledges (TypeScript's onInterrupted fires), or after a short quiet timeout if
the server goes silent. Java's cancelCurrent() blocks until the acknowledgement
(onInterrupted() fires), or up to about 5 seconds. Rust's cancel() and Dart's
cancel() return once the server acknowledges, or after 30 seconds of silence; late
audio of the cancelled turn is discarded. In Rust, drop the audio stream you are
reading before you call cancel(), since both borrow the session. C++'s cancel() returns once the server
acknowledges and is the one session method you may call from another thread while
send() or flush() is blocked. Stop local playback as soon as
you call it: a few in-flight frames may arrive before the acknowledgement. Python's
synchronous session exposes cancel_current() too. See
Barge-in for the full protocol.
Updating settings mid-session
Change generation parameters on a live connection without reconnecting. The update is explicit and acknowledged, and returns the parameters now in effect:
session = client.tts.streaming_session(voice_id=1071, cfg_scale=2.0)
async for chunk in session.send("Default voicing.", flush=True):
play_audio(chunk.audio)
# Change cfg_scale / speed / temperature for the next turn:
effective = await session.update_settings(cfg_scale=1.5, speed=1.1)
# -> {"cfg_scale": 1.5, "temperature": ..., "speed": 1.1, ...}
async for chunk in session.send("Now a bit faster and softer.", flush=True):
play_audio(chunk.audio)const effective = await session.updateSettings({ cfgScale: 1.5, speed: 1.1 });
// -> { cfgScale: 1.5, temperature: ..., speed: 1.1, ... }
session.send('Now a bit faster and softer.', true);import com.kugelaudio.sdk.EffectiveSettings;
import com.kugelaudio.sdk.SettingsUpdate;
EffectiveSettings effective = session.updateSettings(
SettingsUpdate.builder().cfgScale(1.5).speed(1.1).build());
// effective.getCfgScale() == 1.5, effective.getSpeed() == 1.1, ...
session.send("Now a bit faster and softer.", true);use kugelaudio::SettingsUpdate;
let effective = session
.update_settings(&SettingsUpdate::new().cfg_scale(1.5).speed(1.1))
.await?;
// effective.cfg_scale == Some(1.5), effective.speed == Some(1.1), ...
let mut audio = session.send("Now a bit faster and softer.", true);
while let Some(chunk) = audio.next().await {
play_audio(&chunk?.audio);
}kugelaudio::GenerationSettings change;
change.cfg_scale = 1.5;
change.speed = 1.1;
kugelaudio::GenerationSettings effective = session.update_settings(change);
// *effective.cfg_scale == 1.5, *effective.speed == 1.1, ...
session.send("Now a bit faster and softer.", /*flush=*/true, play_chunk);final effective = await session.updateSettings(cfgScale: 1.5, speed: 1.1);
// -> {cfg_scale: 1.5, temperature: ..., speed: 1.1, ...}
await session.flush('Now a bit faster and softer.');Only generation parameters are updatable: CFG scale, temperature, speed, max new tokens, language and normalize. Identity and audio-format settings (voice, model, sample rate, output format, project, dictionary IDs) are fixed for the connection. The change applies to the next turn; call it between turns. Multi-context sessions expose the same update, session-scoped: it applies to contexts started after the update.
update_settings(**params) takes cfg_scale, temperature, speed,
max_new_tokens, language, normalize and returns a dict. Change identity
settings with update_config() after end_session(). The server rejects an
out-of-range value or a non-updatable field with a typed error.
StreamingSessionSync exposes update_settings() too.
updateSettings(settings) takes cfgScale, temperature, speed,
maxNewTokens, language, normalize and resolves with EffectiveSettings.
Change identity settings with updateConfig() after endSession(). The promise
rejects if the server refuses an out-of-range value.
updateSettings(SettingsUpdate) takes .cfgScale, .temperature, .speed,
.maxNewTokens, .language, .normalize and returns EffectiveSettings. To
change identity settings, create a new StreamingSession with a fresh
StreamConfig. updateSettings() throws a KugelAudioException if the server
rejects an out-of-range value.
update_settings(&SettingsUpdate) takes .cfg_scale, .temperature, .speed,
.max_new_tokens, .language, .normalize and returns GenerationSettings, whose
fields are Options. Change identity settings with update_config() between turns.
An empty update or a value the server rejects returns an Error::Validation.
update_settings(GenerationSettings) takes the optional fields cfg_scale,
temperature, speed, max_new_tokens, language, normalize and returns the
GenerationSettings now in effect. Change identity settings with update_config()
between turns. An empty update or a value the server rejects throws
ValidationError.
updateSettings(...) takes the named arguments cfgScale, temperature, speed,
maxNewTokens, language, normalize and returns the server's echo as a
Map<String, Object?>. Change identity settings with updateConfig() between turns.
Calling it without an argument throws an ArgumentError; a value the server rejects
throws a KugelAudioException.
Server-side chunking
The server accumulates tokens and synthesizes each sentence as soon as it ends. Two
session settings tune its buffer; what they do is in
Chunking and latency. The
auto-flush timeout defaults to 500 ms and the maximum buffer to 1000 characters:
| SDK | Auto-flush timeout (ms) | Maximum buffered characters |
|---|---|---|
| Python | flush_timeout_ms (session factory argument) | max_buffer_length (set with session.update_config(...) before the first send) |
| TypeScript | flushTimeoutMs | maxBufferLength |
| Java | .flushTimeoutMs(int) | not available |
| Rust | StreamConfig::flush_timeout_ms(u32) | StreamConfig::max_buffer_length(u32) |
| C++ | StreamConfig::flush_timeout_ms | StreamConfig::max_buffer_length |
| Dart | StreamConfig(flushTimeoutMs: ...) | StreamConfig(maxBufferLength: ...) |
The SDKs also accept the ElevenLabs-style chunk_length_schedule / auto_mode
(Python, and the Rust and C++ StreamConfig), chunkLengthSchedule / autoMode
(TypeScript and Dart) and .chunkLengthSchedule(...) / .autoMode(...) (Java) for
compatibility. They have no effect: the streaming socket ignores both and answers
with an "Unknown fields ignored" warning.
Session reference
A session is created with streaming_session(...) (async) or
streaming_session_sync(...) (sync). Both accept the same configuration:
voice_id, model_id, cfg_scale, temperature, max_new_tokens,
sample_rate, flush_timeout_ms, normalize, language, word_timestamps,
speed, dictionary_ids, project_id, and an on_word_timestamps callback.
Although voice_id defaults to None in the SDK signature, it must be set
before the first synthesis; omission fails with MISSING_VOICE_ID.
project_id and dictionary_ids select dictionaries (see
Dictionaries at synthesis). The
full field list is StreamConfig.
The async StreamingSession exposes:
| Method | Returns | Description |
|---|---|---|
await session.connect() | None | Open and authenticate the WebSocket. Called automatically by the first send() and by async with. |
session.send(text, flush=False) | AsyncIterator[AudioChunk] | Buffer text and yield audio as it is generated. Without flush, it yields only what arrives within a short poll; flush=True forces synthesis of whatever is buffered and waits for its audio. |
session.flush() | AsyncIterator[AudioChunk] | Flush the buffer and yield remaining audio for the current turn. |
session.drain() | AsyncIterator[AudioChunk] | Signal end-of-input and yield every remaining chunk until the server closes the session. Call it before end_session() or close() to receive the tail audio. |
await session.end_session() | dict | End the current session but keep the WebSocket open for reuse. The server flushes buffered text, but audio still in flight is discarded client-side; call drain() first to receive it. |
await session.cancel_current() | None | Barge-in: abandon the current turn and drop buffered or queued text, keeping the socket open. |
session.update_config(config=None, **kwargs) | None | Update configuration (e.g. voice_id) for the next session after end_session(). Keyword arguments change only those fields; passing a whole StreamConfig replaces every field. |
await session.update_settings(**params) | dict | Change generation parameters (cfg_scale, temperature, speed, max_new_tokens, language, normalize) mid-connection and return the effective values. Applies to the next turn. |
await session.close() | dict | Cancel a pending connection attempt, or end the session and close the WebSocket. Like end_session(), it discards audio still in flight. |
session.last_word_timestamps | list[WordTimestamp] | The most recently received word timestamps. |
session.last_final | dict | None | End-of-audio stats from the most recently completed turn: the server's {"final": true, ...} frame (ElevenLabs isFinal equivalent), sent after the turn's last audio frame. None before the first turn completes; not updated on a barge-in cancel. |
session.last_usage | SessionUsage | None | Per-session usage (audio time + amount charged) from the most recently closed session, for billing your own customers per conversation. None before the first session closes. See SessionUsage. |
StreamingSessionSync has a subset of this API without await/async for:
send(), flush(), and drain() return list[AudioChunk], and
cancel_current(), update_settings(), close(), and the
last_word_timestamps / last_final / last_usage properties behave the
same. It has no connect(), end_session(), or update_config(), so session
reuse and voice switching are async-only.
streamingSession(config, callbacks) returns a StreamingSession. config is a
StreamConfig and callbacks a
StreamingSessionCallbacks. voiceId is optional in the
TypeScript declaration but must be set before the first synthesis; omission fails
with MISSING_VOICE_ID. projectId and dictionaryIds select pronunciation
dictionaries (see Dictionaries at synthesis).
| Method | Returns | Description |
|---|---|---|
session.connect() | Promise<void> | Open and authenticate the WebSocket. |
session.send(text, flush?) | void | Buffer text; pass true as flush to force synthesis of whatever is buffered. |
session.cancelCurrent() | Promise<void> | Barge-in: abandon the current turn and drop buffered or queued text, keeping the socket open. |
session.endSession() | Promise<void> | End the current session: the server flushes buffered text and the promise resolves once the remaining audio has been delivered. Keeps the WebSocket open for reuse. |
session.updateConfig(config) | void | Update configuration (e.g. voiceId) for the next session after endSession(). |
session.updateSettings(settings) | Promise<EffectiveSettings> | Change generation parameters (cfgScale, temperature, speed, maxNewTokens, language, normalize) mid-connection; resolves with the effective values. Applies to the next turn. |
session.close() | Promise<void> | End the session like endSession(), then close the WebSocket. |
session.isConnected | boolean | Whether the underlying WebSocket is open. |
session.lastUsage | SessionUsage | null | Per-session usage (audio time + amount charged) from the most recently closed session, for billing your own customers per conversation. null before the first session closes. See SessionUsage. |
There is no drain() or lastFinal in the TypeScript SDK: endSession() and
close() already deliver the tail audio, and the end-of-turn stats arrive in
the onFinal callback.
client.streamingSession(StreamConfig, StreamCallbacks) returns a
StreamingSession (AutoCloseable). StreamConfig.builder() exposes the same
generation knobs as GenerateRequest plus the server-side auto-flush timeout:
StreamConfig config = StreamConfig.builder()
.voiceId(1071)
.modelId("kugel-3.5")
.language("en")
.temperature(0.5)
.speed(1.0)
.flushTimeoutMs(500) // server-side auto-flush timeout
.build();.voiceId(int) takes a numeric voice ID. It is optional in the Java builder but
required before the first synthesis; omission fails with MISSING_VOICE_ID.
| Builder method | Description |
|---|---|
.flushTimeoutMs(int) | Emit buffered text after this many ms of no new input. Default 500. |
.chunkLengthSchedule(List<Integer>), .autoMode(boolean) | Accepted for compatibility, no effect (see Server-side chunking). |
.projectId(long) | Project whose dictionaries apply to the session. See Dictionaries at synthesis. |
.dictionaryIds(List<Integer>) | Per-session dictionary selection; requires .projectId(long). List.of() disables dictionaries. |
.speed(double) | Playback speed 0.8 to 1.2 (pitch-preserving WSOLA). |
.cfgScale, .temperature, .maxNewTokens, .sampleRate, .outputFormat, .normalize, .wordTimestamps | Same meaning as on GenerateRequest. outputFormat accepts pcm_8000, pcm_16000, pcm_22050, pcm_24000, ulaw_8000, or alaw_8000. |
StreamingSession methods: connect(), send(text[, flush]), flush(),
endSession(), cancelCurrent(), updateSettings(SettingsUpdate),
getLastUsage(), and close(). The callbacks are listed under
Callbacks.
client.tts().streaming_session(StreamConfig) returns a StreamingSession.
StreamConfig::new(voice_id) sets the voice; the other fields are builder methods
(.model_id, .cfg_scale, .temperature, .max_new_tokens, .sample_rate,
.output_format, .flush_timeout_ms, .max_buffer_length, .normalize,
.language, .word_timestamps, .speed, .dictionary_ids, .project_id). The
full field list is StreamConfig.
| Method | Returns | Description |
|---|---|---|
session.connect().await | Result<()> | Open the WebSocket. send() connects on its own when needed. |
session.send(text, flush) | stream of Result<AudioChunk> | Send text and yield the audio generated so far. With flush true, the turn ends and the stream yields all its audio. |
session.flush() | stream of Result<AudioChunk> | End the turn: synthesize the buffered text and yield its audio. |
session.drain() | stream of Result<AudioChunk> | Signal end of input and yield the remaining audio until the server closes the turn. The socket stays open. |
session.end_session().await | Result<Option<FinalStats>> | End the turn and keep the socket. Audio still in flight is discarded; call drain() first to receive it. |
session.cancel().await | Result<()> | Barge-in: stop the current turn and drop its buffered text, keeping the socket open. |
session.update_config(config) | () | Replace the StreamConfig for the next turn. |
session.update_settings(&update).await | Result<GenerationSettings> | Change generation parameters mid-connection; applies to the next turn. |
session.close().await | Result<Option<FinalStats>> | End the turn and close the socket. Audio still in flight is discarded. |
session.on_word_timestamps(callback) | () | Call callback every time word timestamps arrive. |
session.last_word_timestamps() | &[WordTimestamp] | The most recently received word timestamps. |
session.last_final() | Option<&FinalStats> | End-of-audio stats of the most recently completed turn. |
session.last_usage() | Option<&SessionUsage> | Usage of the most recently closed turn. See SessionUsage. |
session.is_connected() | bool | Whether the WebSocket is open. |
client.tts().streaming_session(StreamConfig, StreamingSessionCallbacks) returns a
StreamingSession; the callbacks (on_word_timestamps, on_warning) are optional.
StreamConfig is a struct with the fields listed under
StreamConfig; set voice_id before the first synthesis.
| Method | Returns | Description |
|---|---|---|
session.connect() | void | Open the WebSocket, so the first send() skips the handshake. |
session.send(text, flush, on_chunk) | void | Send text and receive the audio it produced so far. A send(text, flush) overload returns std::vector<AudioChunk>. |
session.flush(on_chunk) | void | Flush the buffered text and receive its audio; the turn ends. |
session.drain(on_chunk) | void | Signal end of input and receive every remaining chunk. The next send() starts a fresh turn on the same socket. |
session.end_session() | SessionStats | End the turn and keep the socket. Audio still in flight is discarded. |
session.cancel() | void | Barge-in. Thread-safe: may be called while another thread is inside send() or flush(). |
session.update_config(config) | void | Replace the configuration for the next turn. |
session.update_settings(settings) | GenerationSettings | Change generation parameters mid-connection; applies to the next turn. |
session.close() | SessionStats | End the session and close the WebSocket. |
session.last_word_timestamps() | std::vector<WordTimestamp> | The most recently received word timestamps. |
session.last_final() | std::optional<TurnFinal> | End-of-audio stats of the most recently completed turn. |
session.last_usage() | std::optional<SessionUsage> | Usage of the most recently closed turn. See SessionUsage. |
session.is_connected() | bool | Whether the WebSocket is open. |
Call the methods from one thread at a time, except cancel().
client.tts.streamingSession([StreamConfig]) returns a StreamingSession.
StreamConfig takes named arguments, listed under
StreamConfig; set voiceId before the first synthesis.
| Member | Returns | Description |
|---|---|---|
session.audio | Stream<AudioChunk> | Audio of every turn, in order. Single-subscription; buffers until listened to. |
session.connect() | Future<void> | Open the WebSocket. send() connects on its own when needed. |
session.send(text) | Future<void> | Send text for the current turn; completes once the frame is written. |
session.flush([text]) | Future<TurnResult?> | End the turn and complete once its audio was delivered. null when no turn was in progress or it was cancelled. |
session.speak(tokens) | Future<TurnResult?> | Send a whole Stream<String> as one turn and flush it. |
session.endSession() | Future<TurnResult?> | End the turn gracefully; the socket stays open. |
session.cancel() | Future<void> | Barge-in: stop the current turn and drop its buffered text, keeping the socket open. |
session.updateConfig(config) | void | Replace the StreamConfig for the next turn. |
session.updateSettings(...) | Future<Map<String, Object?>> | Change generation parameters mid-connection; applies to the next turn. |
session.close() | Future<TurnResult?> | Finish the turn in flight, close the socket and the streams. |
session.wordTimestamps | Stream<List<WordTimestamp>> | Word timestamps as they arrive. |
session.warnings | Stream<String> | Non-fatal server advisories, such as a turn ended after 5 seconds without text. |
session.lastWordTimestamps | List<WordTimestamp> | The most recently received word timestamps. |
session.lastFinal | Map<String, Object?>? | End-of-audio frame of the most recently completed turn. |
session.lastUsage | SessionUsage? | Usage of the most recently ended turn. See SessionUsage. |
session.isConnected | bool | Whether the WebSocket is open. |
Per-session usage
For billing your own customers per conversation, every closed session reports a
SessionUsage: the audio time and the actual amount
charged. One-shot requests and multi-context contexts report it too.
async with client.tts.streaming_session(voice_id=1071) as session:
async for chunk in session.send("Hello!", flush=True):
play_audio(chunk.audio)
usage = session.last_usage # None before the first session closes
if usage is not None:
print(f"audio: {usage.audio_seconds:.1f}s")
if usage.cost_available:
print(f"charged: {usage.cost_cents:.2f} {usage.currency} ct")await session.close();
const usage = session.lastUsage; // null before the first session closes
if (usage) {
console.log(`audio: ${usage.audioSeconds.toFixed(1)}s`);
if (usage.costAvailable) {
console.log(`charged: ${usage.costCents!.toFixed(2)} ${usage.currency} ct`);
}
}import com.kugelaudio.sdk.SessionUsage;
SessionUsage usage = session.getLastUsage();
if (usage != null) {
System.out.printf("audio: %.1fs%n", usage.getAudioSeconds());
if (usage.isCostAvailable()) {
System.out.printf("charged: %.2f %s ct%n",
usage.getCostCents(), usage.getCurrency());
}
}// None before the first turn closes
if let Some(usage) = session.last_usage() {
println!("audio: {:.1}s", usage.audio_seconds);
if let Some(cost) = usage.cost_cents {
println!("charged: {:.2} {} ct", cost, usage.currency.as_deref().unwrap_or(""));
}
}// std::nullopt before the first turn closes
if (std::optional<kugelaudio::SessionUsage> usage = session.last_usage()) {
std::cout << "audio: " << usage->audio_seconds << "s\n";
if (usage->cost_available()) {
std::cout << "charged: " << *usage->cost_cents << ' ' << usage->currency.value_or("")
<< " ct\n";
}
}final turn = await session.flush();
final usage = turn?.usage ?? session.lastUsage; // null before the first turn ends
if (usage != null) {
print('audio: ${usage.audioSeconds.toStringAsFixed(1)}s');
if (usage.costAvailable) {
print('charged: ${usage.costCents!.toStringAsFixed(2)} ${usage.currency} ct');
}
}| SDK | Streaming session (most recently closed) | One-shot request | Multi-context context |
|---|---|---|---|
| Python | session.last_usage | AudioResponse.usage (generate()) | session.usage_for(context_id) |
| TypeScript | session.lastUsage | GenerationStats.usage in onFinal (stream()) | session.usageFor(contextId), or the onContextClosed argument |
| Java | session.getLastUsage() | response.getUsage() on the AudioResponse of generate() / stream() | session.getUsageFor(contextId) |
| Rust | session.last_usage() | FinalStats.usage of StreamEvent::Final, or AudioResponse.usage from collect_response() (stream()) | session.usage_for(context_id) |
| C++ | session.last_usage() | AudioResponse::usage of stream() | session.usage_for(context_id) |
| Dart | session.lastUsage, or TurnResult.usage from flush() | (await stream.result).usage, or AudioResponse.usage from collect() (stream()) | session.usageFor(contextId), or the result of closeContext() |
Multi-context sessions
A multi-context session manages up to 20 independent audio-generation contexts over a single WebSocket (see limits). Each context has its own text buffer, voice, and generation queue: useful for multi-speaker conversations, pre-buffering one stream while another plays, or interleaving audio for dynamic dialogue. The concepts are in the Multi-context guide.
async with client.tts.multi_context_session(language="en") as session:
# Create contexts, optionally with different voices
await session.create_context("narrator", voice_id=1071)
await session.create_context("character", voice_id=1072)
# Send text to a specific context
async for chunk in session.send("narrator", "The story begins."):
play_audio(chunk.audio)
async for chunk in session.send("character", "Hello there!", flush=True):
play_audio(chunk.audio)
# Drain remaining audio and close one context
async for chunk in session.close_context("narrator"):
play_audio(chunk.audio)const session = client.tts.createMultiContextSession({
defaultVoiceId: 1071,
language: 'en',
});
// Resolve a promise per context when the server reports it closed
const closed = new Map<string, () => void>();
const whenClosed = (id: string) =>
new Promise<void>((resolve) => closed.set(id, resolve));
await session.connect({
onChunk: (chunk) => {
// chunk.contextId tells you which speaker this audio belongs to
playAudio(chunk.contextId, chunk.audio);
},
onContextClosed: (contextId, usage) => {
// `usage` carries this conversation's audio time + amount charged (EUR cents)
console.log(`${contextId} finished`, usage);
closed.get(contextId)?.();
},
onError: (err, contextId) => console.error(contextId, err),
});
// Create contexts, optionally with different voices
session.createContext('narrator', { voiceId: 1071 });
session.createContext('character', { voiceId: 1072 });
const done = Promise.all([whenClosed('narrator'), whenClosed('character')]);
// Send text to a specific context
session.send('narrator', 'The story begins.');
session.send('character', 'Hello there!', true); // flush
// A graceful close lets each context finish its audio
session.closeContext('narrator');
session.closeContext('character');
await done;
session.close();import com.kugelaudio.sdk.MultiContextConfig;
import com.kugelaudio.sdk.MultiContextSession;
import com.kugelaudio.sdk.MultiContextCallbacks;
import com.kugelaudio.sdk.CreateContextOptions;
import com.kugelaudio.sdk.AudioChunk;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
MultiContextConfig config = MultiContextConfig.builder()
.language("en")
.sampleRate(24000)
.build();
CountDownLatch bothClosed = new CountDownLatch(2);
try (MultiContextSession session = client.multiContextSession(config)) {
session.connect(new MultiContextCallbacks() {
@Override
public void onChunk(String contextId, AudioChunk chunk) {
playAudio(contextId, chunk.getAudio());
}
@Override
public void onContextClosed(String contextId) {
// Per-context usage (audio time + amount charged): each
// context is billed independently.
System.out.println("[" + contextId + "] usage: " + session.getUsageFor(contextId));
bothClosed.countDown();
}
@Override
public void onError(String contextId, com.kugelaudio.sdk.KugelAudioException error) {
System.err.println("[" + contextId + "] Error: " + error.getMessage());
}
});
session.createContext("speaker-a", CreateContextOptions.builder().voiceId(101).build());
session.createContext("speaker-b", CreateContextOptions.builder().voiceId(202).build());
session.send("speaker-a", "Hello from speaker A!", true);
session.send("speaker-b", "And greetings from speaker B!", true);
// Graceful close: queued text finishes, then context_closed arrives.
session.closeContext("speaker-a");
session.closeContext("speaker-b");
// close() cancels every live context, so wait for both before leaving the block.
bothClosed.await(30, TimeUnit.SECONDS);
}use futures_util::StreamExt;
use kugelaudio::MultiContextConfig;
let mut session = client
.tts()
.multi_context_session(MultiContextConfig::default().language("en"));
session.connect().await?;
// Create contexts, optionally with different voices
session.create_context("narrator", Some(1071.into())).await?;
session.create_context("character", Some(1072.into())).await?;
// Send text to a specific context and receive its audio
for (context, text) in [("narrator", "The story begins."), ("character", "Hello there!")] {
let mut audio = session.send(context, text, true, None);
while let Some(chunk) = audio.next().await {
play_audio(&chunk?.audio);
}
}
// A graceful close lets each context finish its audio
for context in ["narrator", "character"] {
let mut tail = session.close_context(context, false);
while let Some(chunk) = tail.next().await {
play_audio(&chunk?.audio);
}
drop(tail);
println!("{context} usage: {:?}", session.usage_for(context));
}
session.close().await;kugelaudio::MultiContextConfig config;
config.language = "en";
kugelaudio::MultiContextSession session = client.tts().multi_context_session(config);
session.connect();
// Create contexts, optionally with different voices
session.create_context("narrator", 1071);
session.create_context("character", 1072);
auto play_chunk = [](const kugelaudio::AudioChunk& chunk) {
// chunk.context_id tells you which speaker this audio belongs to
play_audio(*chunk.context_id, chunk.audio);
};
// Send text to a specific context and receive its audio
session.send("narrator", "The story begins.", /*flush=*/true, play_chunk);
session.send("character", "Hello there!", /*flush=*/true, play_chunk);
// A graceful close lets each context finish its audio
session.close_context("narrator", /*immediate=*/false, play_chunk);
session.close_context("character", /*immediate=*/false, play_chunk);
if (auto usage = session.usage_for("narrator")) {
std::cout << "narrator usage: " << usage->audio_seconds << "s\n";
}
session.close();final session = client.tts.multiContextSession(MultiContextConfig(language: 'en'));
// Audio of every context arrives on one stream; chunk.contextId names the speaker
session.audio.listen((chunk) => playAudio(chunk.contextId!, chunk.audio));
await session.connect();
// Create contexts, optionally with different voices
await session.createContext('narrator', voiceId: 1071);
await session.createContext('character', voiceId: 1072);
// Send text to a specific context; flush() waits for its audio
await session.send('narrator', 'The story begins.');
await session.flush('character', 'Hello there!');
// A graceful close lets each context finish its audio and returns its usage
final narratorUsage = await session.closeContext('narrator');
await session.closeContext('character');
print('narrator usage: ${narratorUsage?.audioSeconds}s');
await session.close();Every context that synthesizes text must have an effective voice; a context without
one fails on its first text with MISSING_VOICE_ID. To barge in on one speaker (the
user talks over speaker A), close only that context immediately: its in-flight
generation and queued text are dropped, and the other contexts and the connection
stay open.
async for chunk in session.close_context("narrator", immediate=True):
play_audio(chunk.audio)session.closeContext('narrator', true); // immediate = truesession.closeContext("speaker-a", true); // immediate = truelet mut tail = session.close_context("narrator", true); // immediate = true
while let Some(chunk) = tail.next().await {
play_audio(&chunk?.audio);
}session.close_context("narrator", /*immediate=*/true, play_chunk);await session.closeContext('narrator', immediate: true);Multi-context reference
Create the session with multi_context_session(...):
| Parameter | Type | Default | Description |
|---|---|---|---|
default_voice_id | int | None | None | Default voice for contexts that don't override it. |
model_id | str | None | None | Model to use. |
sample_rate | int | 24000 | Output sample rate. |
cfg_scale | float | 2.0 | Guidance scale (1.2 to 2.5; values outside are clamped). |
temperature | float | None | None | Sampling variance. |
max_new_tokens | int | 2048 | Maximum tokens per generation. |
normalize | bool | True | Enable text normalization. |
language | str | None | None | Normalization language. |
dictionary_ids | list[int] | None | None | Exact dictionary selection; requires project_id. [] disables dictionaries. |
inactivity_timeout | float | 20.0 | Seconds before an idle context auto-closes. |
word_timestamps | bool | False | Request per-word time alignments for every context. |
on_word_timestamps | Callable[[str, list[WordTimestamp]], None] | None | None | Called with (context_id, timestamps) as alignments arrive. |
project_id | int | None | None | Project whose dictionaries apply; see Dictionaries at synthesis. |
Set default_voice_id on the session or pass voice_id to every
create_context() call.
MultiContextSession methods:
| Method | Returns | Description |
|---|---|---|
await session.connect() | None | Open the WebSocket. Called automatically by async with. |
await session.create_context(context_id, voice_id=None) | None | Create a context with an optional voice override. |
session.send(context_id, text, flush=False, chunk_complete_idle_timeout=None) | AsyncIterator[AudioChunk] | Send text to a context and yield its audio. |
session.flush(context_id) | AsyncIterator[AudioChunk] | Flush a context's buffer. |
session.close_context(context_id, immediate=False) | AsyncIterator[AudioChunk] | Close a context and drain its audio. immediate=True barges in, discarding buffered or queued text. |
await session.keep_alive(context_id) | None | Reset a context's inactivity timeout. |
await session.update_settings(**params) | dict | Change generation parameters for contexts started after the update; returns the effective values. |
session.get_word_timestamps(context_id) | list[WordTimestamp] | The most recently received word timestamps for a context. |
await session.close() | dict | Cancel a pending connection attempt, or close the established session and return stats. |
session.usage_for(context_id) | SessionUsage | None | Per-context usage (audio time + amount charged) for a closed context; each context is its own conversation. None until that context closes. See SessionUsage. |
session.context_usage | dict[str, SessionUsage] | Map of context_id to usage for every context closed so far. |
session.active_contexts | set[str] | The set of currently active context IDs. |
session.session_id | str | None | Server-assigned session ID. |
session.is_alive | bool | Whether the underlying WebSocket is still usable for send(). |
send(), closeContext() and close() return immediately. close() shuts
the socket at once and does not wait for audio, so close each context first and
wait for its onContextClosed, as in the example above.
Create the session with createMultiContextSession(config?):
| Config field | Type | Default | Description |
|---|---|---|---|
defaultVoiceId | number | none | Default voice for contexts that don't override it. |
sampleRate | number | 24000 | Output sample rate. |
outputFormat | string | none | Combined codec + rate token (pcm_8000, pcm_16000, pcm_22050, pcm_24000, ulaw_8000, alaw_8000). |
cfgScale | number | 2.0 | Guidance scale (1.2 to 2.5; values outside are clamped). |
temperature | number | none | Sampling variance (0.0 to 1.0). Omit to use the server default. |
maxNewTokens | number | 2048 | Maximum tokens per generation. |
normalize | boolean | true | Enable text normalization. |
language | string | none | Normalization language. |
projectId | number | none | Project whose pronunciation dictionaries apply. See Dictionaries at synthesis. |
dictionaryIds | number[] | none | Exact dictionary selection; [] disables dictionaries. A non-empty list requires projectId. |
inactivityTimeout | number | 20.0 | Seconds before an idle context auto-closes. |
Unlike the Python SDK, the TypeScript multi-context config has no modelId or
wordTimestamps option. Configure defaultVoiceId or pass voiceId to each
createContext() call.
MultiContextSession methods and properties:
| Member | Returns | Description |
|---|---|---|
connect(callbacks) | Promise<void> | Open the WebSocket with MultiContextCallbacks. |
createContext(contextId, { voiceId? }) | void | Create a context with an optional voice override. The type also accepts voiceSettings; it is kept for compatibility and has no effect. |
send(contextId, text, flush?) | void | Send text to a context. |
flush(contextId) | void | Flush a context's buffer. |
closeContext(contextId, immediate?) | void | Close a context; its remaining audio is delivered before onContextClosed fires. immediate=true barges in, discarding buffered or queued text. |
keepAlive(contextId) | void | Reset a context's inactivity timeout. |
updateSettings(settings) | Promise<EffectiveSettings> | Change generation parameters for contexts started after the update. |
close() | void | Close the session and the WebSocket immediately. Audio still in flight is dropped. |
usageFor(contextId) | SessionUsage | null | Per-context usage (audio time + amount charged) for a closed context; each context is its own conversation. null until that context closes. Also delivered as the second arg to onContextClosed. See SessionUsage. |
contextUsage | Map<string, SessionUsage> | Map of contextId to usage for every context closed so far. |
sessionId | string | null | Server-assigned session ID. |
activeContexts | string[] | Currently active context IDs. |
isConnected | boolean | Whether the WebSocket is open. |
Audio arrives via the onChunk callback as a MultiContextAudioChunk: an
AudioChunk plus a contextId field identifying its context. The interfaces are
in Multi-context types.
MultiContextConfig.builder() accepts .sampleRate(int), .outputFormat(String),
.normalize(boolean), .language(String), .wordTimestamps(boolean),
.temperature(double), .projectId(long), and .dictionaryIds(List<Integer>).
Supported outputFormat tokens are pcm_8000, pcm_16000, pcm_22050,
pcm_24000, ulaw_8000, and alaw_8000.
Per-context overrides are set with CreateContextOptions.builder():
.voiceId(int), .cfgScale(double), and .maxNewTokens(int).
Every Java context that synthesizes text must set .voiceId(int) in its
CreateContextOptions. MultiContextConfig has no default-voice field, so
createContext(id) without options creates a voiceless context whose first text
fails with MISSING_VOICE_ID.
MultiContextSession methods: connect(MultiContextCallbacks),
createContext(id[, options]), send(id, text[, flush]), flush(id),
closeContext(id[, immediate]), keepAlive(id), updateSettings(SettingsUpdate),
getSessionId(), getActiveContexts(), isConnected(), and close().
close() cancels every live context, so wait for each context's
onContextClosed first.
For per-conversation billing, getUsageFor(contextId) returns the
SessionUsage (audio time + amount charged) for a
closed context (each context is its own conversation), or null until it closes;
getContextUsage() returns a snapshot map of contextId to usage for all closed
contexts.
MultiContextCallbacks (only onChunk is required):
| Method | Fires when |
|---|---|
onChunk(String contextId, AudioChunk chunk) | Audio arrives for a context (required). |
onContextCreated(String contextId) | A context is created on the server. |
onGenerationStarted(String contextId) | Generation starts for a context. |
onContextClosed(String contextId) | A context finishes and closes. |
onSessionClosed() | The whole session closes. |
onWordTimestamps(String contextId, List<WordTimestamp> timestamps) | Word timestamps arrive for a context. |
onError(String contextId, KugelAudioException error) | An error occurs. |
Create the session with client.tts().multi_context_session(MultiContextConfig).
MultiContextConfig::default() takes the builder methods .default_voice_id,
.model_id, .sample_rate, .output_format, .cfg_scale, .temperature,
.max_new_tokens, .normalize, .language, .dictionary_ids, .project_id and
.word_timestamps. It has no speed setting. Set .default_voice_id(...) or pass a
voice to every create_context() call.
| Method | Returns | Description |
|---|---|---|
session.connect().await | Result<()> | Open the WebSocket. |
session.create_context(id, voice_id).await | Result<()> | Create a context with an optional VoiceId (Some(1071.into())); None uses the default voice. |
session.send(id, text, flush, chunk_complete_idle) | stream of Result<AudioChunk> | Send text to a context (created on first use) and yield its audio. chunk_complete_idle overrides how long to wait for the next sentence. |
session.flush(id) | stream of Result<AudioChunk> | Flush a context's buffer and yield its audio. |
session.close_context(id, immediate) | stream of Result<AudioChunk> | Close a context and yield its remaining audio. immediate true barges in, discarding buffered or queued text. |
session.keep_alive(id).await | Result<()> | Reset a context's inactivity timeout. |
session.update_settings(&update).await | Result<GenerationSettings> | Change generation parameters for contexts started after the update. |
session.close().await | Option<FinalStats> | Close every context and the WebSocket. |
session.on_word_timestamps(callback) | () | Call callback(context_id, words) every time word timestamps arrive. |
session.word_timestamps(id) | &[WordTimestamp] | The most recently received word timestamps of a context. |
session.usage_for(id) | Option<&SessionUsage> | Usage of a closed context. See SessionUsage. |
session.context_usage() | &HashMap<String, SessionUsage> | Usage of every context closed so far. |
session.active_contexts() | HashSet<String> | Currently active context IDs. |
session.is_alive() | bool | Whether the socket is usable. |
There is no session-wide cancel; close a context with immediate true to barge in.
Create the session with
client.tts().multi_context_session(MultiContextConfig, MultiContextCallbacks).
MultiContextConfig is a struct with default_voice_id, model_id,
sample_rate, output_format, cfg_scale, temperature, max_new_tokens,
normalize, language, dictionary_ids, project_id, word_timestamps and
speed. Set default_voice_id or pass a voice to every create_context() call.
The callbacks hold one optional on_word_timestamps(context_id, stamps).
| Method | Returns | Description |
|---|---|---|
session.connect() | void | Open the WebSocket. |
session.prepare_voice(voice_id) | void | Warm up a voice before any text exists; no context, no charge. |
session.create_context(id, voice_id) | void | Create a context with an optional voice; waits for the server's confirmation. |
session.send(id, text, flush, on_chunk) | void | Send text to a context (created on first use) and receive its audio. A send(id, text, flush) overload returns std::vector<AudioChunk>. |
session.flush(id, on_chunk) | void | Flush a context's buffer and receive its audio. |
session.close_context(id, immediate, on_chunk) | void | Close a context and receive its remaining audio. immediate true barges in. |
session.cancel(id) | void | Barge in on one context, or on every live context when id is omitted. |
session.keep_alive(id) | void | Reset a context's inactivity timeout. |
session.update_settings(settings) | GenerationSettings | Change generation parameters for contexts started after the update. |
session.close() | std::optional<double> | Close every context and the WebSocket; returns the session's audio seconds when reported. |
session.word_timestamps(id) | std::vector<WordTimestamp> | The most recently received word timestamps of a context. |
session.usage_for(id) | std::optional<SessionUsage> | Usage of a closed context. See SessionUsage. |
session.context_usage() | std::map<std::string, SessionUsage> | Usage of every context closed so far. |
session.active_contexts() | std::set<std::string> | Currently active context IDs. |
session.is_alive() | bool | Whether the WebSocket is open. |
Audio of one context that arrives while you wait on another is kept and returned by that context's next call. Call the methods from one thread at a time.
Create the session with client.tts.multiContextSession([MultiContextConfig]).
MultiContextConfig takes the named arguments defaultVoiceId, modelId,
sampleRate, outputFormat, cfgScale, temperature, maxNewTokens,
normalize, language, speed, dictionaryIds, projectId and
wordTimestamps. Set defaultVoiceId or pass voiceId to every
createContext() call.
| Member | Returns | Description |
|---|---|---|
session.audio | Stream<AudioChunk> | Audio of every context in arrival order; chunk.contextId names the context. |
session.connect() | Future<void> | Open the WebSocket. Call it again after a dropped connection. |
session.createContext(id, voiceId: ...) | Future<void> | Create a context and wait for the server's confirmation. |
session.send(id, text, flush: false) | Future<void> | Send text to a context (created on first use). |
session.flush(id, [text]) | Future<void> | Flush a context and complete once its audio was delivered. |
session.closeContext(id, immediate: false) | Future<SessionUsage?> | Close a context and complete with its usage. immediate: true barges in. |
session.keepAlive(id) | Future<void> | Reset a context's inactivity timeout. |
session.updateSettings(...) | Future<Map<String, Object?>> | Change generation parameters for contexts started after the update. |
session.close() | Future<Map<String, Object?>> | Close every context and the socket; returns the server's session_closed frame. |
session.wordTimestamps | Stream<ContextWordTimestamps> | Word timestamps as they arrive, with their contextId. |
session.wordTimestampsFor(id) | List<WordTimestamp> | The most recently received word timestamps of a context. |
session.usageFor(id) | SessionUsage? | Usage of a closed context. See SessionUsage. |
session.contextUsage | Map<String, SessionUsage> | Usage of every context closed so far. |
session.activeContexts | Set<String> | Currently active context IDs. |
session.isConnected | bool | Whether the WebSocket is open. |
There is no session-wide cancel; close a context with immediate: true to barge in.
Word timestamps in streaming
Word timestamps work with one-shot streams and streaming sessions. They add no extra audio latency: they arrive shortly after the corresponding audio chunk (see Latency) and are useful for barge-in handling, subtitle synchronization, and lip-sync.
One-shot stream:
from kugelaudio import AudioChunk, WordTimestamp
# Timestamps are yielded as list[WordTimestamp] items between audio chunks
for item in client.tts.stream(
text="Hello, how are you today?",
model_id="kugel-3.5",
voice_id=1071,
word_timestamps=True,
):
if isinstance(item, AudioChunk):
play_audio(item.audio)
elif isinstance(item, list) and item and isinstance(item[0], WordTimestamp):
for ts in item:
print(f"{ts.word}: {ts.start_ms}-{ts.end_ms}ms")await client.tts.stream(
{
text: 'Hello, how are you today?',
modelId: 'kugel-3.5',
voiceId: 1071,
wordTimestamps: true,
},
{
onChunk: (chunk) => {
playAudio(chunk.audio);
},
onWordTimestamps: (timestamps) => {
for (const ts of timestamps) {
console.log(`${ts.word}: ${ts.startMs}-${ts.endMs}ms`);
}
},
onFinal: (stats) => {
console.log(`Done: ${stats.durationMs}ms`);
},
}
);client.tts().stream(
GenerateRequest.builder("Hello, how are you today?")
.modelId("kugel-3.5")
.voiceId(1071)
.language("en")
.wordTimestamps(true)
.build(),
new StreamCallbacks() {
@Override
public void onChunk(AudioChunk chunk) {
playAudio(chunk.getAudio());
}
@Override
public void onWordTimestamps(java.util.List<WordTimestamp> timestamps) {
for (WordTimestamp ts : timestamps) {
System.out.printf("%s: %dms-%dms%n",
ts.getWord(), ts.getStartMs(), ts.getEndMs());
}
}
}
);// Timestamps arrive as StreamEvent::WordTimestamps between audio events
let request = TtsRequest::new("Hello, how are you today?")
.model_id("kugel-3.5")
.voice_id(1071)
.word_timestamps(true);
let mut stream = client.tts().stream(&request).await?;
while let Some(event) = stream.next().await {
match event? {
StreamEvent::Audio(chunk) => play_audio(&chunk.audio),
StreamEvent::WordTimestamps(words) => {
for ts in words {
println!("{}: {}-{}ms", ts.word, ts.start_ms, ts.end_ms);
}
}
StreamEvent::Final(stats) => println!("Done: {:?}ms", stats.duration_ms),
_ => {}
}
}kugelaudio::GenerateRequest request;
request.text = "Hello, how are you today?";
request.model_id = "kugel-3.5";
request.voice_id = 1071;
request.word_timestamps = true;
kugelaudio::StreamCallbacks callbacks;
callbacks.on_audio = [](const kugelaudio::AudioChunk& chunk) { play_audio(chunk.audio); };
callbacks.on_word_timestamps = [](const std::vector<kugelaudio::WordTimestamp>& stamps) {
for (const auto& ts : stamps) {
std::cout << ts.word << ": " << ts.start_ms << "-" << ts.end_ms << "ms\n";
}
};
kugelaudio::AudioResponse result = client.tts().stream(request, callbacks);
// result.word_timestamps holds all of themfinal stream = client.tts.stream(
text: 'Hello, how are you today?',
modelId: 'kugel-3.5',
voiceId: 1071,
wordTimestamps: true,
onWordTimestamps: (timestamps) {
for (final ts in timestamps) {
print('${ts.word}: ${ts.startMs}-${ts.endMs}ms');
}
},
);
await for (final chunk in stream) {
playAudio(chunk.audio);
}
print('Done: ${(await stream.result).durationMs}ms');
// stream.wordTimestamps holds all of themStreaming session: timestamps are delivered per chunk after the corresponding audio data.
async with client.tts.streaming_session(
voice_id=1071,
word_timestamps=True,
) as session:
async for chunk in session.send("Hello, how are you today?"):
play_audio(chunk.audio)
async for chunk in session.flush():
play_audio(chunk.audio)
# Access the latest word timestamps
timestamps = session.last_word_timestamps
for ts in timestamps:
print(f"{ts.word}: {ts.start_ms}ms - {ts.end_ms}ms (score: {ts.score:.2f})")
# Or register a callback to process timestamps as they arrive
def on_timestamps(timestamps):
for ts in timestamps:
print(f" {ts.word} [{ts.start_ms}-{ts.end_ms}ms]")
async with client.tts.streaming_session(
voice_id=1071,
on_word_timestamps=on_timestamps,
) as session:
async for chunk in session.send("Hello world!"):
play_audio(chunk.audio)
async for chunk in session.flush():
play_audio(chunk.audio)const session = client.tts.streamingSession(
{ voiceId: 1071, wordTimestamps: true },
{
onChunk: (chunk) => playAudio(chunk.audio),
onWordTimestamps: (timestamps) => {
for (const ts of timestamps) {
console.log(` ${ts.word} [${ts.startMs}-${ts.endMs}ms]`);
}
},
}
);
await session.connect();
session.send('Hello, how are you today?');
await session.close();StreamConfig config = StreamConfig.builder()
.voiceId(1071)
.wordTimestamps(true)
.build();
try (StreamingSession session = client.streamingSession(config, new StreamCallbacks() {
@Override
public void onChunk(AudioChunk chunk) {
playAudio(chunk.getAudio());
}
@Override
public void onWordTimestamps(java.util.List<WordTimestamp> timestamps) {
for (WordTimestamp ts : timestamps) {
System.out.printf(" %s [%d-%dms]%n", ts.getWord(), ts.getStartMs(), ts.getEndMs());
}
}
})) {
session.send("Hello, how are you today?");
}let mut session = client
.tts()
.streaming_session(StreamConfig::new(1071).word_timestamps(true));
// Register a callback to process timestamps as they arrive
session.on_word_timestamps(|timestamps| {
for ts in timestamps {
println!(" {} [{}-{}ms]", ts.word, ts.start_ms, ts.end_ms);
}
});
let mut audio = session.send("Hello, how are you today?", true);
while let Some(chunk) = audio.next().await {
play_audio(&chunk?.audio);
}
drop(audio);
// Or read the latest word timestamps
for ts in session.last_word_timestamps() {
println!("{}: {}ms - {}ms", ts.word, ts.start_ms, ts.end_ms);
}
session.close().await?;kugelaudio::StreamConfig config;
config.voice_id = 1071;
config.word_timestamps = true;
kugelaudio::StreamingSessionCallbacks callbacks;
callbacks.on_word_timestamps = [](const std::vector<kugelaudio::WordTimestamp>& stamps) {
for (const auto& ts : stamps) {
std::cout << " " << ts.word << " [" << ts.start_ms << "-" << ts.end_ms << "ms]\n";
}
};
auto session = client.tts().streaming_session(config, callbacks);
session.send("Hello, how are you today?", /*flush=*/true,
[](const kugelaudio::AudioChunk& chunk) { play_audio(chunk.audio); });
// session.last_word_timestamps() holds the latest frame
session.close();final session = client.tts.streamingSession(
StreamConfig(voiceId: 1071, wordTimestamps: true),
);
session.audio.listen((chunk) => playAudio(chunk.audio));
session.wordTimestamps.listen((timestamps) {
for (final ts in timestamps) {
print(' ${ts.word} [${ts.startMs}-${ts.endMs}ms]');
}
});
final turn = await session.flush('Hello, how are you today?');
// turn?.wordTimestamps holds the turn's timestamps
await session.close();Multi-context sessions deliver timestamps per context: on_word_timestamps and
get_word_timestamps(context_id) in Python, onWordTimestamps(contextId, timestamps)
in Java, on_word_timestamps and word_timestamps(context_id) in Rust,
MultiContextCallbacks::on_word_timestamps and word_timestamps(context_id) in
C++, and the wordTimestamps stream and wordTimestampsFor(contextId) in Dart. The
TypeScript multi-context session has no word-timestamp option.
Reconnecting after a server restart
During a rolling deploy the server closes sockets with code 1012 or 1013 (see
Errors for the close codes).
Sessions retry silently when no audio has been delivered yet. When the session
cannot replay the turn without repeating audio, it raises
ServerRestartingError (a ConnectionError subclass): reconnect and resend the
in-flight turn after retry_after seconds. See Errors.
The SDK replays transparently where it can and calls onServerRestart instead of
failing: a one-shot stream() that had no audio yet re-issues the request once after
error.retryAfter seconds; a streaming session reconnects and replays the current
turn; a multi-context session reconnects and replays pending text. When it cannot
replay without repeating audio, it fails with ServerRestartingError: reconnect and
resend the in-flight turn after retryAfter seconds.
The Java SDK treats any close code outside 4xxx as a normal end of stream: no
error callback fires, and it does not reconnect or replay the turn. The next
send() on that session fails. Open a new session and resend the current turn.
Streaming and multi-context sessions reconnect and replay a turn once while it has
produced no audio yet (a multi-context session re-creates its contexts first). When
audio of the turn was already delivered, the call fails with
Error::ServerRestarting: reconnect and resend the in-flight turn after
retry_after() seconds.
Streaming and multi-context sessions reconnect and replay a turn once while it has
produced no audio yet (a multi-context session re-creates its contexts first). When
audio of the turn was already delivered, they throw ServerRestartingError (a
ConnectionError): reconnect and resend the in-flight turn after retry_after()
seconds. Audio received before the error is complete up to the last chunk.
A one-shot stream() that had no audio yet is replayed once on a fresh socket.
Streaming and multi-context sessions reconnect and replay a turn once while it has
produced no audio yet. When audio of the turn was already delivered, the turn fails
with ServerRestartingException: resend it after retryAfter.
Next steps
- Types and errors:
AudioChunk,StreamConfig, callbacks,SessionUsage,WordTimestamp - Text normalization: languages and spell tags in streaming
- Barge-in: the full interruption protocol