Pipecat Integration
Use KugelAudio speech to text and text to speech in Pipecat voice AI pipelines with the official Python services: setup, server-side end of turn, configuration, runtime voice switching and noise removal.
On this page
KugelAudio provides official STT and TTS services for Pipecat: KugelAudioSTTService transcribes the user and decides on the server when they have finished their turn, and KugelAudioTTSService speaks the replies.
Why Use KugelAudio with Pipecat?
- Native services: Drop-in
STTServiceandTTSServicefor Pipecat pipelines - Server-side end of turn: KugelTurn, the end-of-turn model behind the STT, hears when the user has finished, instead of a fixed silence timer
- Persistent WebSocket: Connection reuse keeps the handshake off the hot path
- Built-in metrics: Automatic TTFB (Pipecat's name for time-to-first-audio) and usage metrics
- Low latency: streaming TTS built for real-time agents. See Latency for how to measure TTFA on your deployment.
Installation
pip install "kugelaudio[pipecat]"This installs the KugelAudio SDK with its Pipecat dependency
(pipecat-ai>=0.0.62 on Python 3.11 or newer). Both services support
Pipecat 0.x and 1.x.
Quick Start
A complete local voice bot for Pipecat 1.x: microphone in, KugelAudio STT, OpenAI LLM, KugelAudio TTS, speaker out. Install the Pipecat services it uses:
pip install "kugelaudio[pipecat]" "pipecat-ai[local,openai]"Save this as bot.py:
import asyncio
import os
from pipecat.frames.frames import TTSSpeakFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.local.audio import (
LocalAudioTransport,
LocalAudioTransportParams,
)
from pipecat.turns.user_turn_strategies import ExternalUserTurnStrategies
from kugelaudio.pipecat import KugelAudioSTTService, KugelAudioTTSService
async def main():
transport = LocalAudioTransport(
LocalAudioTransportParams(
audio_in_enabled=True,
audio_out_enabled=True,
audio_out_sample_rate=24000,
)
)
# The server hears when the user has finished; no local VAD needed.
stt = KugelAudioSTTService(language="en")
llm = OpenAILLMService(
api_key=os.environ["OPENAI_API_KEY"],
settings=OpenAILLMService.Settings(
model="gpt-4o-mini",
system_instruction="You are a helpful voice assistant. Keep answers short.",
),
)
tts = KugelAudioTTSService(
model="kugel-3.5",
voice_id=1071,
sample_rate=24000, # must match the transport's output rate
language="en", # normalization language (default: the voice's)
)
tts.prewarm() # needs the running event loop, so call it inside main()
context_aggregator = LLMContextAggregatorPair(
LLMContext(),
# Let the STT's turns drive the conversation.
user_params=LLMUserAggregatorParams(
user_turn_strategies=ExternalUserTurnStrategies()
),
)
pipeline = Pipeline(
[
transport.input(),
stt,
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
]
)
task = PipelineTask(pipeline)
await task.queue_frames([TTSSpeakFrame("Hi! How can I help you?")])
await PipelineRunner().run(task)
if __name__ == "__main__":
asyncio.run(main())Set KUGELAUDIO_API_KEY and OPENAI_API_KEY, then run python bot.py. Use
headphones: the local audio transport has no echo cancellation, so speaker
output can feed back into the microphone. For Pipecat 0.x, or another
transport or LLM, keep the stt and tts lines and build the rest with the
APIs of your Pipecat version.
Speech to text and turns
KugelAudioSTTService keeps one streaming connection open for the call and
sends the user's audio as it arrives, pauses included. KugelTurn, the
end-of-turn model on the server, decides when the user has finished, and the
service turns that into Pipecat's turn frames: the turn starts, interim
transcripts follow while a long turn is spoken, then the final
TranscriptionFrame and the end of the turn. A short turn arrives as one final
transcript. ExternalUserTurnStrategies tells the user aggregator to follow
these turns instead of its own VAD and turn analyzer; from Pipecat 1.11 the
service recommends it by itself, and passing it explicitly does no harm.
When a turn starts, the service interrupts the bot, so the user can talk over it. Because the server only reports a short turn when it ends, a brief "stop" interrupts once the user has paused, not on the first syllable.
- Language. Pass one code such as
"de", a list such as["de", "fr"]to detect among, or nothing to detect German or English. Speech to text covers German, English, French, Italian, Spanish, Portuguese, Dutch, Polish and Romanian. To switch mid-call, pushSTTUpdateSettingsFrame(delta=STTSettings(language=Language.FR))(frompipecat.services.settings; Pipecat before 0.0.104 takessettings={"language": Language.FR}). The service reconnects, and a turn in progress is dropped. - Turn length.
turn_detection_level="conservative"holds longer through a pause mid-thought, so fewer turns are cut off early. - Names and terms.
boosted_phrasesand your project's dictionaries make names, brands and domain words come out in your spelling.
The API key needs the Speech to text permission.
Configuration
TTS service parameters
Keyword arguments of KugelAudioTTSService(...).
| Parameter | Type | Default | Description |
|---|---|---|---|
api_key | str | KUGELAUDIO_API_KEY env | Your KugelAudio API key |
model | str | kugel-3.5 | TTS model (kugel-3.5 or kugel-3), see Models |
voice_id | int | required | Voice ID to use for synthesis |
sample_rate | int | 24000 | Output sample rate in Hz |
cfg_scale | float | 2.0 | CFG scale, clamped to [1.2, 2.5] |
max_new_tokens | int | 2048 | Maximum tokens to generate (1 to 2048) |
language | str | None | None | ISO 639-1 language code (e.g., en, de). Unset: recognized per sentence, falling back to the voice's primary language |
normalize | bool | True | Apply text normalization |
project_id | int | None | None | Project whose pronunciation dictionaries apply. Unset: no dictionary applies |
dictionary_ids | list[int] | None | None | Dictionaries by ID; [] disables them. A non-empty list requires project_id |
region | str | None | None | Set "eu" for the direct EU endpoint; overridden by base_url |
base_url | str | https://api.kugelaudio.com | API base URL |
on_word_timestamps | Callable[[str, list[WordTimestamp]], None] | None | None | Callback receiving the context ID and each batch of word timestamps. Setting it turns word timestamps on, see Word timestamps |
STT service parameters
Keyword arguments of KugelAudioSTTService(...). Every one is optional; one
left unset is not sent, so the server default applies.
| Parameter | Type | Default | Description |
|---|---|---|---|
api_key | str | KUGELAUDIO_API_KEY env | Your KugelAudio API key. An eu- prefix selects the EU endpoint |
language | str | list[str] | None | None | A code such as "de", a list of codes to detect among, or "auto". Unset: German or English, detected |
turn_detection | str | None | None | Who ends a turn: "kugelturn" (server default), "vad" (a fixed pause) or "manual" |
turn_detection_level | str | None | None | With "kugelturn": "responsive" (server default) or "conservative" |
boosted_phrases | list[str] | None | None | Names, brands and domain terms to spell as written |
project_id | int | None | None | Project whose dictionaries apply. Unset: the API key's project |
dictionary_ids | list[int] | None | None | Dictionaries to apply; [] applies none. Unset: every active dictionary of the project |
model | str | None | None | Speech-to-text model ID. Unset: the API's default model |
sample_rate | int | 16000 | Rate in Hz the audio is sent at; pipeline audio at another rate is resampled to it |
region | str | None | None | Set "eu" for the direct EU endpoint; overridden by base_url |
base_url | str | https://api.kugelaudio.com | API base URL |
Supported TTS sample rates
| Rate | Notes |
|---|---|
24000 | Native rate (recommended) |
22050 | CD quality |
16000 | Wideband telephony |
8000 | Narrowband telephony |
Lower rates are resampled server-side. Any other value raises ValueError in
the constructor.
Performance
Set language when you know it (Text normalization), and call
prewarm() inside your async setup code (it needs a running event loop and
does nothing otherwise) so the first reply does not pay the WebSocket
handshake. The service then reuses that connection across run_tts() calls
and reconnects transparently if it drops. On Pipecat 1.x each assistant turn
still opens a new server-side context; the service creates it when the LLM
starts responding, so the setup overlaps the LLM's time-to-first-token. The
KugelAudio TTFA: log line measures text send to first audio chunk on the
WebSocket, without LLM or STT time. See Latency for how to measure
end to end and Chunking & per-segment latency
for how the server chunks text.
Usage Patterns
Updating the Voice at Runtime
# Switch voice mid-conversation (closes the cached WebSocket first)
await tts.set_voice("300")Pipeline Frame Flow
The KugelAudioTTSService emits standard Pipecat frames, in this order:
TTSStartedFrame: audio generation has begunTTSAudioRawFrame: raw PCM audio chunks (16-bit, mono), one or moreTTSStoppedFrame: audio generation is completeErrorFrame: if an error occurs during synthesis
Metrics Support
The service reports Pipecat's TTFB metric (Pipecat's name for time-to-first-audio,
measured from request to first audio chunk) and TTS usage (characters per
request) automatically; tts.can_generate_metrics() returns True.
Removing Background Noise
KugelAudioEnhanceFilter puts KugelAudio
speech enhancement (Clarity, clarity-1) on
the transport's audio input, the audio_in_filter slot Pipecat uses for noise
filters. It removes background noise before VAD and STT hear the user. Pass
speaker= with a clean 2 to 8 second sample of one person and it keeps only
that voice.
The Quick Start bot with enhancement on the microphone input. Save it as
bot.py, set the same two API keys, and run python bot.py:
import asyncio
import os
from pipecat.frames.frames import TTSSpeakFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.local.audio import (
LocalAudioTransport,
LocalAudioTransportParams,
)
from pipecat.turns.user_turn_strategies import ExternalUserTurnStrategies
from kugelaudio import load_audio
from kugelaudio.pipecat import (
KugelAudioEnhanceFilter,
KugelAudioSTTService,
KugelAudioTTSService,
)
async def main():
transport = LocalAudioTransport(
LocalAudioTransportParams(
audio_in_enabled=True,
audio_in_filter=KugelAudioEnhanceFilter(), # reads KUGELAUDIO_API_KEY
# keep one voice: KugelAudioEnhanceFilter(speaker=load_audio("caller.wav"))
audio_out_enabled=True,
audio_out_sample_rate=24000,
)
)
# The server hears when the user has finished; no local VAD needed.
stt = KugelAudioSTTService(language="en")
llm = OpenAILLMService(
api_key=os.environ["OPENAI_API_KEY"],
settings=OpenAILLMService.Settings(
model="gpt-4o-mini",
system_instruction="You are a helpful voice assistant. Keep answers short.",
),
)
tts = KugelAudioTTSService(
model="kugel-3.5", voice_id=1071, sample_rate=24000, language="en"
)
tts.prewarm()
context_aggregator = LLMContextAggregatorPair(
LLMContext(),
# Let the STT's turns drive the conversation.
user_params=LLMUserAggregatorParams(
user_turn_strategies=ExternalUserTurnStrategies()
),
)
pipeline = Pipeline(
[
transport.input(),
stt,
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
]
)
task = PipelineTask(pipeline)
await task.queue_frames([TTSSpeakFrame("Hi! How can I help you?")])
await PipelineRunner().run(task)
if __name__ == "__main__":
asyncio.run(main())Any Pipecat transport takes the filter the same way, as
audio_in_filter= in its TransportParams. Push FilterEnableFrame(False)
to switch it off mid-call (the stream closes and audio passes through) and
FilterEnableFrame(True) to switch it back on.
Before you turn it on:
- Latency. Enhancement runs on KugelAudio's servers, not on the device, so the audio makes a round trip over the network and the bot hears the user somewhat later than with an on-device filter. Frames keep their size; they arrive delayed.
- Plan and limits. Enhancement uses your organization's Clarity plan: the free month, then paid. Every second of input audio is enhanced, silence included, and each running pipeline holds one stream open, so on the free month its concurrency limit caps how many sessions can be enhanced at once. See Free month and paid plan.
- Connection. The filter opens its connection and readies enhancement on the server when the pipeline starts, before the first audio, and closes the connection when the pipeline stops. After 60 seconds without audio the server closes it and the next audio reconnects.
- Failures. While the stream connects (after a quiet period enhancement can
take several seconds to start; the filter waits up to 30 seconds), or
reconnects after a rate limit or a network error, the bot hears the original
audio and a warning is logged for a failure. A
rejected API key, spent credits, an ended free month or an invalid request turn
enhancement off for the rest of the session with one error log; pass
on_error=to be told.
| Parameter | Default | Description |
|---|---|---|
api_key | KUGELAUDIO_API_KEY env | API key. |
client | None | An existing KugelAudio client, instead of api_key and base_url. |
model | "clarity-1" | Enhancement model. |
speaker | None | load_audio(...) sample of the one voice to keep. Unset: remove noise, keep every voice. |
max_backlog_s | 2.0 | Seconds the server may fall behind before the original audio is used and the stream reconnects. |
on_error | None | Called once with the error that turned enhancement off. |
base_url | https://api.kugelaudio.com | API base URL. |
Environment Variables
| Variable | Required | Description |
|---|---|---|
KUGELAUDIO_API_KEY | Yes | Your KugelAudio API key, used by the STT and the TTS |
OPENAI_API_KEY | Yes* | Required if using OpenAI LLM |
Troubleshooting
The STT is refused with 403
Two causes: speech to text is not enabled for your organization yet (the message
says Transcription is not enabled for this organization.), or the API key lacks
the Speech to text permission.
Ask us to enable it, or create a key with the permission.
Turns do not follow the STT
Before Pipecat 1.11 the user aggregator uses its own VAD and turn analyzer
unless told otherwise. Pass user_turn_strategies=ExternalUserTurnStrategies()
as in the Quick Start.
API key not found
Make sure KUGELAUDIO_API_KEY is set in your environment or pass api_key directly:
tts = KugelAudioTTSService(api_key="your-api-key", voice_id=1071)Unsupported sample rate error
The TTS service accepts 24000, 22050, 16000 and 8000. Set the same rate on
the service and on your transport's audio output (in the Quick Start,
sample_rate=24000 and audio_out_sample_rate=24000).
WebSocket connection fails
Verify your base_url is correct and the KugelAudio API is reachable. The service connects via WebSocket (wss://) for audio streaming. If a persistent connection drops, the service automatically reconnects on the next run_tts() call.
High latency
Check Performance first: language unset and prewarm() not
called (or called outside the event loop) are the usual causes. Then measure
from the same region as the endpoint, see
Measuring TTFA correctly.
Python version incompatibility
The Pipecat integration requires Python 3.11 or newer. Check your version:
python --version