Skip to content
KugelAudioDocs

LiveKit Integration

Add KugelAudio speech to text and text to speech to a LiveKit Agents voice agent in Python or TypeScript: install the plugins, server-side end of turn, tune the voice, word timestamps and noise removal.

On this page

KugelAudio ships official STT and TTS plugins for LiveKit Agents in Python and in TypeScript. They plug into AgentSession as the stt and tts providers. The STT streams the caller's audio and its server decides when the caller has finished their turn; the TTS keeps one WebSocket open per agent, so every reply streams over a warm connection.

1. Install

Bash
pip install "kugelaudio[livekit]" livekit-plugins-openai livekit-plugins-silero

The livekit extra installs livekit-agents. The OpenAI and Silero plugins are only needed for the LLM and VAD in the example below; swap them for the providers you use.

2. Add it to your agent

Pass the plugins as stt and tts, set the session's turn detection to "stt" so the end of each caller turn comes from KugelAudio speech to text, and call prewarm() while the call is being set up, so the first response does not pay for the connection. Save the Python version as voice_agent.py or the TypeScript version as voice_agent.ts.

Python
from livekit.agents import Agent, AgentSession, JobContext
from livekit.agents import WorkerOptions, cli
from livekit.plugins import openai, silero
from kugelaudio.livekit import STT as KugelAudioSTT
from kugelaudio.livekit import TTS as KugelAudioTTS

async def entrypoint(ctx: JobContext):
    await ctx.connect()

    tts = KugelAudioTTS(model="kugel-3.5", voice_id=1071, language="en")
    tts.prewarm()

    session = AgentSession(
        stt=KugelAudioSTT(language="en"),
        turn_handling={"turn_detection": "stt"},
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=tts,
        vad=silero.VAD.load(),
    )
    await session.start(
        room=ctx.room,
        agent=Agent(instructions="You are a helpful voice assistant."),
    )
    await session.say("Hello! How can I help you?")

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
Bash
python voice_agent.py console   # local test
python voice_agent.py start     # production worker

The worker reads KUGELAUDIO_API_KEY, LIVEKIT_URL, LIVEKIT_API_KEY and LIVEKIT_API_SECRET from the environment (full list under Environment variables). Pick a voice in Voices; the plugin has no default voice.

How the STT hears the caller

The STT plugin streams the caller's audio, pauses included, to KugelAudio speech to text over one WebSocket per session. KugelTurn, the end-of-turn model on the server, decides when the caller has finished, and the plugin reports that turn to LiveKit as a final transcript and the end of speech. With the session's turn detection set to "stt" the agent answers on that decision instead of its own turn detector. Keep the Silero VAD: LiveKit still uses it to notice the caller talking over the agent. LiveKit Agents releases without turn_handling take the same setting as turn_detection="stt" (turnDetection: 'stt').

  • Language. Pass one code such as "de", a list such as ["de", "fr"] to detect among, or nothing to detect German or English. Speech to text covers German, English, French, Italian, Spanish, Portuguese, Dutch, Polish and Romanian.
  • Turn length. turn_detection_level="conservative" holds longer through a pause mid-thought, so fewer turns are cut off early.
  • Interim transcripts arrive while a long turn is still being spoken; a short turn arrives as one final transcript.
  • Names and terms. boosted_phrases and your project's dictionaries make names, brands and domain words come out in your spelling.

The API key needs the Speech to text permission. Every option is under STT options.

3. Tune the sound

speed, temperature and cfg_scale shape the audio. Set them in the constructor or change them mid-call; either way the next turn picks up the new values.

Python
tts = KugelAudioTTS(
    model="kugel-3.5",
    voice_id=1071,
    language="de",
    speed=1.1,          # 0.8 to 1.2
    temperature=0.3,    # 0.0 (most stable) to 1.0 (most varied)
    cfg_scale=2.0,      # 1.2 to 2.5
)

# Later, e.g. from a tool call: switch voice, slow down
tts.update_options(voice_id=300, speed=0.9)

What each parameter does is documented in Generation parameters. Two rules specific to the plugin:

  • Out-of-range values raise instead of being clamped: speed outside 0.8 to 1.2 or temperature outside 0.0 to 1.0 is rejected before anything is applied, so a bad update leaves the current settings untouched.
  • speed and temperature are session-wide. All turns share one socket and the server applies the last value it received. A change therefore takes effect for turns started after it; a turn already playing keeps the old value.

4. Word timestamps (optional)

Turn on the word-timestamps option (off by default, see TTS options) and the server aligns each audio chunk and delivers per-word timing alongside it. LiveKit uses this for barge-in and transcript sync. Timestamps arrive after their audio chunk, so playback never waits for them. Details in Word timestamps.

5. Remove background noise (optional)

enhancement() puts KugelAudio speech enhancement (Clarity, clarity-1) on the agent's audio input, the slot LiveKit uses for noise cancellation. It removes background noise before VAD and STT hear the user. Pass speaker= with a clean 2 to 8 second sample of one person and it keeps only that voice. Python only; it needs livekit-agents 1.3.10 or later. On an older release, importing enhancement raises an ImportError that says to upgrade:

Bash
pip install -U "kugelaudio[livekit]" "livekit-agents>=1.3.10" \
  livekit-plugins-openai livekit-plugins-silero

The agent from step 2, with enhancement on the participant's audio:

Python
from livekit.agents import Agent, AgentSession, JobContext
from livekit.agents import WorkerOptions, cli, room_io
from livekit.plugins import openai, silero
from kugelaudio import load_audio
from kugelaudio.livekit import STT as KugelAudioSTT
from kugelaudio.livekit import TTS as KugelAudioTTS
from kugelaudio.livekit import enhancement

async def entrypoint(ctx: JobContext):
    await ctx.connect()

    tts = KugelAudioTTS(model="kugel-3.5", voice_id=1071, language="en")
    tts.prewarm()

    session = AgentSession(
        stt=KugelAudioSTT(language="en"),
        turn_handling={"turn_detection": "stt"},
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=tts,
        vad=silero.VAD.load(),
    )
    await session.start(
        room=ctx.room,
        agent=Agent(instructions="You are a helpful voice assistant."),
        room_options=room_io.RoomOptions(
            audio_input=room_io.AudioInputOptions(
                noise_cancellation=enhancement(),  # reads KUGELAUDIO_API_KEY
                # keep one voice: enhancement(speaker=load_audio("caller.wav"))
            ),
        ),
    )
    await session.say("Hello! How can I help you?")

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

LiveKit releases that still take room_input_options= accept the same processor as room_io.RoomInputOptions(noise_cancellation=enhancement()). The input must be mono, which is LiveKit's default; on a multi-channel input enhancement turns itself off with an error log.

Before you turn it on:

  • Latency. Enhancement runs on KugelAudio's servers, not on the device, so the audio makes a round trip over the network and the agent hears the user somewhat later than with an on-device filter. Frames keep their size; they arrive delayed.
  • Plan and limits. Enhancement uses your organization's Clarity plan: the free month, then paid. Every second of the participant's audio is enhanced, silence included, and each running agent session holds one stream open, so on the free month its concurrency limit caps how many sessions can be enhanced at once. See Free month and paid plan.
  • Connection. The processor opens its connection and readies enhancement on the server when the agent starts, before the participant speaks, and keeps the connection for the participant's later tracks and after a sample-rate change. After 60 seconds without audio the server closes it and the next audio reconnects.
  • Failures. While the stream connects (after a quiet period enhancement can take several seconds to start; the processor waits up to 30 seconds), or reconnects after a rate limit or a network error, the agent hears the original audio and a warning is logged for a failure. A rejected API key, spent credits, an ended free month or an invalid request turn enhancement off for the rest of the session with one error log; pass on_error= to be told.
OptionDefaultDescription
api_keyKUGELAUDIO_API_KEY envAPI key.
clientNoneAn existing KugelAudio client, instead of api_key and base_url.
model"clarity-1"Enhancement model.
speakerNoneload_audio(...) sample of the one voice to keep. Unset: remove noise, keep every voice.
max_backlog_s2.0Seconds the server may fall behind before the original audio is used and the stream reconnects.
on_errorNoneCalled once with the error that turned enhancement off.
base_urlhttps://api.kugelaudio.comAPI base URL.

6. Move over step by step (optional)

Your LiveKit deployment, your rooms and your telephony stay as they are. Each step below works on its own and can be rolled back by changing one argument, so you can stop wherever it suits you.

Change the voice. Replace the tts argument of your AgentSession with KugelAudioTTS.

Change speech to text. Replace the stt argument with KugelAudioSTT and set turn_detection to "stt", as in Add it to your agent. Keep your VAD for barge-in.

Dispatch a room to one agent with a dispatch rule or through the API:

Python
await lk.agent_dispatch.create_dispatch(
    api.CreateAgentDispatchRequest(agent_name="support", room=room_name)
)

Keep the transcript

Every line of the call reaches conversation_item_added. Handoffs between agents arrive there too, so keep only messages:

Python
@session.on("conversation_item_added")
def on_item(event: ConversationItemAddedEvent) -> None:
    if isinstance(event.item, ChatMessage):
        store(event.item.role, event.item.text_content)

Speech to speech with Fluid 1

Fluid 1 hears the caller, decides when to answer and speaks, so one model takes the place of speech to text, the LLM and text to speech. LiveKit's OpenAI plugin connects to it with GPTLiveModel:

Python
import os

from livekit.plugins.openai.realtime import GPTLiveModel

model = GPTLiveModel(
    api_key=os.environ["KUGELAUDIO_API_KEY"],
    base_url="https://api.kugelaudio.com/v1",
    model="fluid-1",
    voice="973",
    delegation="client",
)
  • Set delegation="client". Your agent answers each delegation_created from its own code. The plugin's default, a hosted back office, is not tested with KugelAudio yet.
  • voice takes a numeric KugelAudio voice id.
  • Tested with version 1.8.2 of the plugin.

Sessions, delegation, interruptions and limits are on the speech to speech pages.

Reference

TTS options

Keyword arguments of kugelaudio.livekit.TTS(...); all of them except api_key, region, base_url, http_session and sample_rate can be changed later with update_options(...).

OptionDefaultDescription
api_keyKUGELAUDIO_API_KEY envAPI key. An eu- prefix selects the EU endpoint.
voice_idNoneVoice ID. Required before the first turn.
model"kugel-3.5"TTS model.
languageNoneISO 639-1 code such as "de". Unset: recognized per sentence, falling back to the voice's primary language.
speedNonePlayback speed, 0.8 to 1.2. Unset means the server default 1.0.
temperatureNoneSampling variance, 0.0 to 1.0. Unset means the engine default.
dictionary_idsNonePronunciation dictionaries by ID; [] disables them. A non-empty list requires project_id.
project_idNoneProject whose dictionaries apply. Unset: no dictionary applies.
cfg_scale2.0Voice adherence, clamped to 1.2 to 2.5.
max_new_tokens2048Maximum tokens per turn, 1 to 2048.
sample_rate24000Output rate in Hz, see Sample rates.
normalizeTrueText normalization, see Text processing.
word_timestampsFalsePer-word timing, see above.
regionNone"eu" for the direct EU endpoint. Overridden by base_url.
base_urlhttps://api.kugelaudio.comAPI base URL.
http_sessionNoneaiohttp ClientSession to reuse.

The plugin can also be registered under LiveKit's plugin namespace:

Python
from kugelaudio.livekit import register_plugin
register_plugin()

from livekit.plugins import kugelaudio
tts = kugelaudio.TTS(model="kugel-3.5", voice_id=1071)

STT options

Every option is optional. One left unset is not sent, so the server default applies.

Keyword arguments of kugelaudio.livekit.STT(...).

OptionDefaultDescription
api_keyKUGELAUDIO_API_KEY envAPI key. An eu- prefix selects the EU endpoint.
languageNoneA code such as "de", a list of codes to detect among, or "auto". Unset: German or English, detected.
turn_detectionNoneWho ends a turn: "kugelturn" (server default), "vad" (a fixed pause) or "manual".
turn_detection_levelNoneWith "kugelturn": "responsive" (server default) or "conservative".
boosted_phrasesNoneNames, brands and domain terms to spell as written.
project_idNoneProject whose dictionaries apply. Unset: the API key's project.
dictionary_idsNoneDictionaries to apply; [] applies none. Unset: every active dictionary of the project.
modelNoneSpeech-to-text model ID. Unset: the API's default model.
sample_rate16000Rate in Hz the audio is resampled to before it is sent.
regionNone"eu" for the direct EU endpoint. Overridden by base_url.
base_urlhttps://api.kugelaudio.comAPI base URL.

turn_detection here is the STT's own setting, sent to the server. The session's turn_handling={"turn_detection": "stt"} is a separate setting: it tells LiveKit to take the turns from the STT.

Sample rates

For the TTS output, 24000 Hz is the native rate and the right choice unless the transport needs another one. 22050, 16000, 8000 and 44100 are resampled server-side; use 8000 or 16000 for telephony.

Environment variables

VariableRequiredDescription
KUGELAUDIO_API_KEYYesYour KugelAudio API key
LIVEKIT_URLYesYour LiveKit server URL
LIVEKIT_API_KEYYesLiveKit API key
LIVEKIT_API_SECRETYesLiveKit API secret
OPENAI_API_KEYIf using OpenAI LLMOpenAI key

Using the plugin without AgentSession

Direct synthesize() and stream() calls

AgentSession drives these for you. Call them directly only when you consume the audio frames yourself.

Python
# One-shot
async for event in tts.synthesize("Hello, this is a complete sentence."):
    ...  # event.frame is a LiveKit AudioFrame

# Streaming, e.g. from an LLM token stream
stream = tts.stream()
stream.push_text("Hello, ")
stream.push_text("how are you today?")
stream.flush()
stream.end_input()
async for event in stream:
    ...  # event.frame

Troubleshooting

MISSING_VOICE_ID

The plugin has no default voice. Pass a voice ID in the constructor or set it at runtime before the first turn. Browse voices in Voices.

API key not found

Set KUGELAUDIO_API_KEY in the environment or pass the API key option to the constructor.

The STT is refused with 403

Two causes: speech to text is not enabled for your organization yet (the message says Transcription is not enabled for this organization.), or the API key lacks the Speech to text permission. Ask us to enable it, or create a key with the permission.

The agent answers too early or too late

With the session's turn detection set to "stt", the turn ends when KugelAudio speech to text decides the caller has finished. If callers get cut off mid-thought, pass turn_detection_level="conservative" to the STT. Without it, LiveKit's own turn detector or VAD timer decides instead.

First response is slow
  • Call prewarm() during call setup, not right before the first turn
  • Create one TTS instance per worker and reuse it; do not construct one per phrase
  • Measure from the same region as the API before changing synthesis parameters, see Latency
Speed or temperature change did not apply

Both are session-wide: a change binds for turns started after it, and a turn already playing finishes with the old value. Values outside the accepted range raise instead of being clamped, so check for an exception at the call site.

WebSocket connection fails

Check the base URL option and that the API is reachable from the worker over wss://. After aclose() (Python) or close() (TypeScript), create a new instance for the next session.

No word timestamps arrive

Word timestamps are off by default, so check that the option is on. If it is, alignment failed server-side for that chunk: the server logs it and skips the timestamps, and the audio is unaffected.

Next steps