LiveKit Integration
Add KugelAudio speech to text and text to speech to a LiveKit Agents voice agent in Python or TypeScript: install the plugins, server-side end of turn, tune the voice, word timestamps and noise removal.
On this page
KugelAudio ships official STT and TTS plugins for LiveKit Agents
in Python and in TypeScript. They plug into AgentSession as the stt and tts
providers. The STT streams the caller's audio and its server decides when the
caller has finished their turn; the TTS keeps one WebSocket open per agent, so
every reply streams over a warm connection.
1. Install
pip install "kugelaudio[livekit]" livekit-plugins-openai livekit-plugins-sileroThe livekit extra installs livekit-agents. The OpenAI and Silero plugins are
only needed for the LLM and VAD in the example below; swap them for the
providers you use.
npm install kugelaudio @livekit/agents @livekit/rtc-node \
@livekit/agents-plugin-openai @livekit/agents-plugin-silero@livekit/agents and @livekit/rtc-node are optional peer dependencies; the core
kugelaudio client never loads them. The two @livekit/agents-plugin-* packages
are only needed for the LLM and VAD in the example below.
2. Add it to your agent
Pass the plugins as stt and tts, set the session's turn detection to "stt" so
the end of each caller turn comes from KugelAudio speech to text, and call prewarm() while the
call is being set up, so the first response does not pay for the connection. Save the
Python version as voice_agent.py or the TypeScript version as voice_agent.ts.
from livekit.agents import Agent, AgentSession, JobContext
from livekit.agents import WorkerOptions, cli
from livekit.plugins import openai, silero
from kugelaudio.livekit import STT as KugelAudioSTT
from kugelaudio.livekit import TTS as KugelAudioTTS
async def entrypoint(ctx: JobContext):
await ctx.connect()
tts = KugelAudioTTS(model="kugel-3.5", voice_id=1071, language="en")
tts.prewarm()
session = AgentSession(
stt=KugelAudioSTT(language="en"),
turn_handling={"turn_detection": "stt"},
llm=openai.LLM(model="gpt-4o-mini"),
tts=tts,
vad=silero.VAD.load(),
)
await session.start(
room=ctx.room,
agent=Agent(instructions="You are a helpful voice assistant."),
)
await session.say("Hello! How can I help you?")
if __name__ == "__main__":
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))python voice_agent.py console # local test
python voice_agent.py start # production workerimport { fileURLToPath } from 'node:url';
import { WorkerOptions, cli, defineAgent, voice } from '@livekit/agents';
import * as openai from '@livekit/agents-plugin-openai';
import * as silero from '@livekit/agents-plugin-silero';
import { STT as KugelAudioSTT, TTS as KugelAudioTTS } from 'kugelaudio/livekit';
export default defineAgent({
entry: async (ctx) => {
await ctx.connect();
const tts = new KugelAudioTTS({
model: 'kugel-3.5',
voiceId: 1071,
language: 'en',
});
tts.prewarm();
const session = new voice.AgentSession({
stt: new KugelAudioSTT({ language: 'en' }),
turnHandling: { turnDetection: 'stt' },
llm: new openai.LLM({ model: 'gpt-4o-mini' }),
tts,
vad: await silero.VAD.load(),
});
await session.start({
agent: new voice.Agent({
instructions: 'You are a helpful voice assistant.',
}),
room: ctx.room,
});
await session.say('Hello! How can I help you?');
},
});
cli.runApp(new WorkerOptions({ agent: fileURLToPath(import.meta.url) }));npx tsx voice_agent.ts console # local test
npx tsx voice_agent.ts start # production workerThe worker reads KUGELAUDIO_API_KEY, LIVEKIT_URL, LIVEKIT_API_KEY and
LIVEKIT_API_SECRET from the environment (full list under
Environment variables). Pick a voice in
Voices; the plugin has no default voice.
How the STT hears the caller
The STT plugin streams the caller's audio, pauses included, to KugelAudio speech to
text over one WebSocket per session. KugelTurn, the end-of-turn model on the server,
decides when the caller has finished, and the plugin reports that turn to LiveKit as
a final transcript and the end of speech. With the session's turn detection set to
"stt" the agent answers on that decision instead of its own turn detector. Keep
the Silero VAD: LiveKit still uses it to notice the caller talking over the agent.
LiveKit Agents releases without turn_handling take the same setting as
turn_detection="stt" (turnDetection: 'stt').
- Language. Pass one code such as
"de", a list such as["de", "fr"]to detect among, or nothing to detect German or English. Speech to text covers German, English, French, Italian, Spanish, Portuguese, Dutch, Polish and Romanian. - Turn length.
turn_detection_level="conservative"holds longer through a pause mid-thought, so fewer turns are cut off early. - Interim transcripts arrive while a long turn is still being spoken; a short turn arrives as one final transcript.
- Names and terms.
boosted_phrasesand your project's dictionaries make names, brands and domain words come out in your spelling.
The API key needs the Speech to text permission. Every option is under STT options.
3. Tune the sound
speed, temperature and cfg_scale shape the audio. Set them in the constructor or
change them mid-call; either way the next turn picks up the new values.
tts = KugelAudioTTS(
model="kugel-3.5",
voice_id=1071,
language="de",
speed=1.1, # 0.8 to 1.2
temperature=0.3, # 0.0 (most stable) to 1.0 (most varied)
cfg_scale=2.0, # 1.2 to 2.5
)
# Later, e.g. from a tool call: switch voice, slow down
tts.update_options(voice_id=300, speed=0.9)const tts = new KugelAudioTTS({
model: 'kugel-3.5',
voiceId: 1071,
language: 'de',
speed: 1.1, // 0.8 to 1.2
temperature: 0.3, // 0.0 (most stable) to 1.0 (most varied)
cfgScale: 2.0, // 1.2 to 2.5
});
// Later, e.g. from a tool call: switch voice, slow down
tts.updateOptions({ voiceId: 300, speed: 0.9 });What each parameter does is documented in Generation parameters. Two rules specific to the plugin:
- Out-of-range values raise instead of being clamped:
speedoutside0.8to1.2ortemperatureoutside0.0to1.0is rejected before anything is applied, so a bad update leaves the current settings untouched. speedandtemperatureare session-wide. All turns share one socket and the server applies the last value it received. A change therefore takes effect for turns started after it; a turn already playing keeps the old value.
4. Word timestamps (optional)
Turn on the word-timestamps option (off by default, see TTS options) and the server aligns each audio chunk and delivers per-word timing alongside it. LiveKit uses this for barge-in and transcript sync. Timestamps arrive after their audio chunk, so playback never waits for them. Details in Word timestamps.
5. Remove background noise (optional)
enhancement() puts KugelAudio speech enhancement
(Clarity, clarity-1) on the agent's audio input, the slot LiveKit uses for
noise cancellation. It removes background noise before VAD and STT hear the
user. Pass speaker= with a clean 2 to 8 second sample of one person and it
keeps only that voice. Python only; it needs livekit-agents 1.3.10 or later.
On an older release, importing enhancement raises an ImportError that says
to upgrade:
pip install -U "kugelaudio[livekit]" "livekit-agents>=1.3.10" \
livekit-plugins-openai livekit-plugins-sileroThe agent from step 2, with enhancement on the participant's audio:
from livekit.agents import Agent, AgentSession, JobContext
from livekit.agents import WorkerOptions, cli, room_io
from livekit.plugins import openai, silero
from kugelaudio import load_audio
from kugelaudio.livekit import STT as KugelAudioSTT
from kugelaudio.livekit import TTS as KugelAudioTTS
from kugelaudio.livekit import enhancement
async def entrypoint(ctx: JobContext):
await ctx.connect()
tts = KugelAudioTTS(model="kugel-3.5", voice_id=1071, language="en")
tts.prewarm()
session = AgentSession(
stt=KugelAudioSTT(language="en"),
turn_handling={"turn_detection": "stt"},
llm=openai.LLM(model="gpt-4o-mini"),
tts=tts,
vad=silero.VAD.load(),
)
await session.start(
room=ctx.room,
agent=Agent(instructions="You are a helpful voice assistant."),
room_options=room_io.RoomOptions(
audio_input=room_io.AudioInputOptions(
noise_cancellation=enhancement(), # reads KUGELAUDIO_API_KEY
# keep one voice: enhancement(speaker=load_audio("caller.wav"))
),
),
)
await session.say("Hello! How can I help you?")
if __name__ == "__main__":
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))LiveKit releases that still take room_input_options= accept the same processor
as room_io.RoomInputOptions(noise_cancellation=enhancement()). The input must be
mono, which is LiveKit's default; on a multi-channel input enhancement turns
itself off with an error log.
Before you turn it on:
- Latency. Enhancement runs on KugelAudio's servers, not on the device, so the audio makes a round trip over the network and the agent hears the user somewhat later than with an on-device filter. Frames keep their size; they arrive delayed.
- Plan and limits. Enhancement uses your organization's Clarity plan: the free month, then paid. Every second of the participant's audio is enhanced, silence included, and each running agent session holds one stream open, so on the free month its concurrency limit caps how many sessions can be enhanced at once. See Free month and paid plan.
- Connection. The processor opens its connection and readies enhancement on the server when the agent starts, before the participant speaks, and keeps the connection for the participant's later tracks and after a sample-rate change. After 60 seconds without audio the server closes it and the next audio reconnects.
- Failures. While the stream connects (after a quiet period enhancement can
take several seconds to start; the processor waits up to 30 seconds), or
reconnects after a rate limit or a network error, the agent hears the original
audio and a warning is logged for a failure. A
rejected API key, spent credits, an ended free month or an invalid request turn
enhancement off for the rest of the session with one error log; pass
on_error=to be told.
| Option | Default | Description |
|---|---|---|
api_key | KUGELAUDIO_API_KEY env | API key. |
client | None | An existing KugelAudio client, instead of api_key and base_url. |
model | "clarity-1" | Enhancement model. |
speaker | None | load_audio(...) sample of the one voice to keep. Unset: remove noise, keep every voice. |
max_backlog_s | 2.0 | Seconds the server may fall behind before the original audio is used and the stream reconnects. |
on_error | None | Called once with the error that turned enhancement off. |
base_url | https://api.kugelaudio.com | API base URL. |
6. Move over step by step (optional)
Your LiveKit deployment, your rooms and your telephony stay as they are. Each step below works on its own and can be rolled back by changing one argument, so you can stop wherever it suits you.
Change the voice. Replace the tts argument of your AgentSession with KugelAudioTTS.
Change speech to text. Replace the stt argument with KugelAudioSTT and set
turn_detection to "stt", as in Add it to your agent. Keep your
VAD for barge-in.
Dispatch a room to one agent with a dispatch rule or through the API:
await lk.agent_dispatch.create_dispatch(
api.CreateAgentDispatchRequest(agent_name="support", room=room_name)
)Keep the transcript
Every line of the call reaches conversation_item_added. Handoffs between agents arrive
there too, so keep only messages:
@session.on("conversation_item_added")
def on_item(event: ConversationItemAddedEvent) -> None:
if isinstance(event.item, ChatMessage):
store(event.item.role, event.item.text_content)Speech to speech with Fluid 1
Fluid 1 hears the caller, decides when to answer and speaks, so one model takes the place of
speech to text, the LLM and text to speech. LiveKit's OpenAI plugin connects to it with
GPTLiveModel:
import os
from livekit.plugins.openai.realtime import GPTLiveModel
model = GPTLiveModel(
api_key=os.environ["KUGELAUDIO_API_KEY"],
base_url="https://api.kugelaudio.com/v1",
model="fluid-1",
voice="973",
delegation="client",
)- Set
delegation="client". Your agent answers eachdelegation_createdfrom its own code. The plugin's default, a hosted back office, is not tested with KugelAudio yet. voicetakes a numeric KugelAudio voice id.- Tested with version 1.8.2 of the plugin.
Sessions, delegation, interruptions and limits are on the speech to speech pages.
Reference
TTS options
Keyword arguments of kugelaudio.livekit.TTS(...); all of them except api_key,
region, base_url, http_session and sample_rate can be changed later with
update_options(...).
| Option | Default | Description |
|---|---|---|
api_key | KUGELAUDIO_API_KEY env | API key. An eu- prefix selects the EU endpoint. |
voice_id | None | Voice ID. Required before the first turn. |
model | "kugel-3.5" | TTS model. |
language | None | ISO 639-1 code such as "de". Unset: recognized per sentence, falling back to the voice's primary language. |
speed | None | Playback speed, 0.8 to 1.2. Unset means the server default 1.0. |
temperature | None | Sampling variance, 0.0 to 1.0. Unset means the engine default. |
dictionary_ids | None | Pronunciation dictionaries by ID; [] disables them. A non-empty list requires project_id. |
project_id | None | Project whose dictionaries apply. Unset: no dictionary applies. |
cfg_scale | 2.0 | Voice adherence, clamped to 1.2 to 2.5. |
max_new_tokens | 2048 | Maximum tokens per turn, 1 to 2048. |
sample_rate | 24000 | Output rate in Hz, see Sample rates. |
normalize | True | Text normalization, see Text processing. |
word_timestamps | False | Per-word timing, see above. |
region | None | "eu" for the direct EU endpoint. Overridden by base_url. |
base_url | https://api.kugelaudio.com | API base URL. |
http_session | None | aiohttp ClientSession to reuse. |
The plugin can also be registered under LiveKit's plugin namespace:
from kugelaudio.livekit import register_plugin
register_plugin()
from livekit.plugins import kugelaudio
tts = kugelaudio.TTS(model="kugel-3.5", voice_id=1071)Fields of the object passed to new TTS({...}) from kugelaudio/livekit; all of
them except apiKey, baseURL and sampleRate can be changed later with
updateOptions({...}) (a sampleRate passed there is ignored).
| Option | Default | Description |
|---|---|---|
apiKey | KUGELAUDIO_API_KEY env | API key. An eu- prefix selects the EU endpoint. |
voiceId | null | Voice ID. Required before the first turn. |
model | 'kugel-3.5' | TTS model. |
language | unset | ISO 639-1 code such as 'de'. Unset: recognized per sentence, falling back to the voice's primary language. |
speed | unset | Playback speed, 0.8 to 1.2. Unset means the server default 1.0. |
temperature | unset | Sampling variance, 0.0 to 1.0. Unset means the engine default. |
cfgScale | 2.0 | Voice adherence, clamped to 1.2 to 2.5. |
maxNewTokens | 2048 | Maximum tokens per turn, 1 to 2048. |
sampleRate | 24000 | Output rate in Hz, see Sample rates. |
normalize | true | Text normalization, see Text processing. |
wordTimestamps | false | Per-word timing, see above. |
dictionaryIds | unset | Pronunciation dictionaries by ID. Requires projectId. |
projectId | unset | Project the dictionaries belong to. |
baseURL | https://api.kugelaudio.com | API base URL. |
STT options
Every option is optional. One left unset is not sent, so the server default applies.
Keyword arguments of kugelaudio.livekit.STT(...).
| Option | Default | Description |
|---|---|---|
api_key | KUGELAUDIO_API_KEY env | API key. An eu- prefix selects the EU endpoint. |
language | None | A code such as "de", a list of codes to detect among, or "auto". Unset: German or English, detected. |
turn_detection | None | Who ends a turn: "kugelturn" (server default), "vad" (a fixed pause) or "manual". |
turn_detection_level | None | With "kugelturn": "responsive" (server default) or "conservative". |
boosted_phrases | None | Names, brands and domain terms to spell as written. |
project_id | None | Project whose dictionaries apply. Unset: the API key's project. |
dictionary_ids | None | Dictionaries to apply; [] applies none. Unset: every active dictionary of the project. |
model | None | Speech-to-text model ID. Unset: the API's default model. |
sample_rate | 16000 | Rate in Hz the audio is resampled to before it is sent. |
region | None | "eu" for the direct EU endpoint. Overridden by base_url. |
base_url | https://api.kugelaudio.com | API base URL. |
Fields of the object passed to new STT({...}) from kugelaudio/livekit.
| Option | Default | Description |
|---|---|---|
apiKey | KUGELAUDIO_API_KEY env | API key. An eu- prefix selects the EU endpoint. |
language | unset | A code such as 'de', a list of codes to detect among, or 'auto'. Unset: German or English, detected. |
turnDetection | unset | Who ends a turn: 'kugelturn' (server default), 'vad' (a fixed pause) or 'manual'. |
turnDetectionLevel | unset | With 'kugelturn': 'responsive' (server default) or 'conservative'. |
boostedPhrases | unset | Names, brands and domain terms to spell as written. |
projectId | unset | Project whose dictionaries apply. Unset: the API key's project. |
dictionaryIds | unset | Dictionaries to apply; [] applies none. Unset: every active dictionary of the project. |
model | unset | Speech-to-text model ID. Unset: the API's default model. |
sampleRate | 16000 | Rate in Hz the audio is resampled to before it is sent. |
region | unset | 'eu' for the direct EU endpoint. Overridden by baseURL. |
baseURL | https://api.kugelaudio.com | API base URL. |
turn_detection here is the STT's own setting, sent to the server. The session's
turn_handling={"turn_detection": "stt"} is a separate setting: it tells LiveKit to
take the turns from the STT.
Sample rates
For the TTS output, 24000 Hz is the native rate and the right choice unless the
transport needs another one. 22050, 16000, 8000 and 44100 are resampled server-side; use
8000 or 16000 for telephony.
Environment variables
| Variable | Required | Description |
|---|---|---|
KUGELAUDIO_API_KEY | Yes | Your KugelAudio API key |
LIVEKIT_URL | Yes | Your LiveKit server URL |
LIVEKIT_API_KEY | Yes | LiveKit API key |
LIVEKIT_API_SECRET | Yes | LiveKit API secret |
OPENAI_API_KEY | If using OpenAI LLM | OpenAI key |
Using the plugin without AgentSession
Direct synthesize() and stream() calls
AgentSession drives these for you. Call them directly only when you consume the
audio frames yourself.
# One-shot
async for event in tts.synthesize("Hello, this is a complete sentence."):
... # event.frame is a LiveKit AudioFrame
# Streaming, e.g. from an LLM token stream
stream = tts.stream()
stream.push_text("Hello, ")
stream.push_text("how are you today?")
stream.flush()
stream.end_input()
async for event in stream:
... # event.frame// One-shot
for await (const event of tts.synthesize('Hello, this is a complete sentence.')) {
// event.frame is a LiveKit AudioFrame
}
// Streaming, e.g. from an LLM token stream
const stream = tts.stream();
stream.pushText('Hello, ');
stream.pushText('how are you today?');
stream.flush();
stream.endInput();
for await (const event of stream) {
// event.frame
}Troubleshooting
MISSING_VOICE_ID
The plugin has no default voice. Pass a voice ID in the constructor or set it at runtime before the first turn. Browse voices in Voices.
API key not found
Set KUGELAUDIO_API_KEY in the environment or pass the API key option to the
constructor.
The STT is refused with 403
Two causes: speech to text is not enabled for your organization yet (the message
says Transcription is not enabled for this organization.), or the API key lacks
the Speech to text permission.
Ask us to enable it, or create a key with the permission.
The agent answers too early or too late
With the session's turn detection set to "stt", the turn ends when KugelAudio
speech to text decides the caller has finished. If callers get cut off
mid-thought, pass turn_detection_level="conservative" to the STT. Without it,
LiveKit's own turn detector or VAD timer decides instead.
First response is slow
- Call
prewarm()during call setup, not right before the first turn - Create one
TTSinstance per worker and reuse it; do not construct one per phrase - Measure from the same region as the API before changing synthesis parameters, see Latency
Speed or temperature change did not apply
Both are session-wide: a change binds for turns started after it, and a turn already playing finishes with the old value. Values outside the accepted range raise instead of being clamped, so check for an exception at the call site.
WebSocket connection fails
Check the base URL option and that the API is reachable from the worker over
wss://. After aclose() (Python) or close() (TypeScript), create a new
instance for the next session.
No word timestamps arrive
Word timestamps are off by default, so check that the option is on. If it is, alignment failed server-side for that chunk: the server logs it and skips the timestamps, and the audio is unaffected.