Skip to content
KugelAudioDocs

Pipecat Integration

Use KugelAudio speech to text and text to speech in Pipecat voice AI pipelines with the official Python services: setup, server-side end of turn, configuration, runtime voice switching and noise removal.

On this page

KugelAudio provides official STT and TTS services for Pipecat: KugelAudioSTTService transcribes the user and decides on the server when they have finished their turn, and KugelAudioTTSService speaks the replies.

Why Use KugelAudio with Pipecat?

  • Native services: Drop-in STTService and TTSService for Pipecat pipelines
  • Server-side end of turn: KugelTurn, the end-of-turn model behind the STT, hears when the user has finished, instead of a fixed silence timer
  • Persistent WebSocket: Connection reuse keeps the handshake off the hot path
  • Built-in metrics: Automatic TTFB (Pipecat's name for time-to-first-audio) and usage metrics
  • Low latency: streaming TTS built for real-time agents. See Latency for how to measure TTFA on your deployment.

Installation

Bash
pip install "kugelaudio[pipecat]"

This installs the KugelAudio SDK with its Pipecat dependency (pipecat-ai>=0.0.62 on Python 3.11 or newer). Both services support Pipecat 0.x and 1.x.

Quick Start

A complete local voice bot for Pipecat 1.x: microphone in, KugelAudio STT, OpenAI LLM, KugelAudio TTS, speaker out. Install the Pipecat services it uses:

Bash
pip install "kugelaudio[pipecat]" "pipecat-ai[local,openai]"

Save this as bot.py:

Python
import asyncio
import os

from pipecat.frames.frames import TTSSpeakFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.local.audio import (
    LocalAudioTransport,
    LocalAudioTransportParams,
)
from pipecat.turns.user_turn_strategies import ExternalUserTurnStrategies

from kugelaudio.pipecat import KugelAudioSTTService, KugelAudioTTSService


async def main():
    transport = LocalAudioTransport(
        LocalAudioTransportParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
            audio_out_sample_rate=24000,
        )
    )
    # The server hears when the user has finished; no local VAD needed.
    stt = KugelAudioSTTService(language="en")
    llm = OpenAILLMService(
        api_key=os.environ["OPENAI_API_KEY"],
        settings=OpenAILLMService.Settings(
            model="gpt-4o-mini",
            system_instruction="You are a helpful voice assistant. Keep answers short.",
        ),
    )
    tts = KugelAudioTTSService(
        model="kugel-3.5",
        voice_id=1071,
        sample_rate=24000,  # must match the transport's output rate
        language="en",      # normalization language (default: the voice's)
    )
    tts.prewarm()  # needs the running event loop, so call it inside main()

    context_aggregator = LLMContextAggregatorPair(
        LLMContext(),
        # Let the STT's turns drive the conversation.
        user_params=LLMUserAggregatorParams(
            user_turn_strategies=ExternalUserTurnStrategies()
        ),
    )
    pipeline = Pipeline(
        [
            transport.input(),
            stt,
            context_aggregator.user(),
            llm,
            tts,
            transport.output(),
            context_aggregator.assistant(),
        ]
    )

    task = PipelineTask(pipeline)
    await task.queue_frames([TTSSpeakFrame("Hi! How can I help you?")])
    await PipelineRunner().run(task)


if __name__ == "__main__":
    asyncio.run(main())

Set KUGELAUDIO_API_KEY and OPENAI_API_KEY, then run python bot.py. Use headphones: the local audio transport has no echo cancellation, so speaker output can feed back into the microphone. For Pipecat 0.x, or another transport or LLM, keep the stt and tts lines and build the rest with the APIs of your Pipecat version.

Speech to text and turns

KugelAudioSTTService keeps one streaming connection open for the call and sends the user's audio as it arrives, pauses included. KugelTurn, the end-of-turn model on the server, decides when the user has finished, and the service turns that into Pipecat's turn frames: the turn starts, interim transcripts follow while a long turn is spoken, then the final TranscriptionFrame and the end of the turn. A short turn arrives as one final transcript. ExternalUserTurnStrategies tells the user aggregator to follow these turns instead of its own VAD and turn analyzer; from Pipecat 1.11 the service recommends it by itself, and passing it explicitly does no harm.

When a turn starts, the service interrupts the bot, so the user can talk over it. Because the server only reports a short turn when it ends, a brief "stop" interrupts once the user has paused, not on the first syllable.

  • Language. Pass one code such as "de", a list such as ["de", "fr"] to detect among, or nothing to detect German or English. Speech to text covers German, English, French, Italian, Spanish, Portuguese, Dutch, Polish and Romanian. To switch mid-call, push STTUpdateSettingsFrame(delta=STTSettings(language=Language.FR)) (from pipecat.services.settings; Pipecat before 0.0.104 takes settings={"language": Language.FR}). The service reconnects, and a turn in progress is dropped.
  • Turn length. turn_detection_level="conservative" holds longer through a pause mid-thought, so fewer turns are cut off early.
  • Names and terms. boosted_phrases and your project's dictionaries make names, brands and domain words come out in your spelling.

The API key needs the Speech to text permission.

Configuration

TTS service parameters

Keyword arguments of KugelAudioTTSService(...).

ParameterTypeDefaultDescription
api_keystrKUGELAUDIO_API_KEY envYour KugelAudio API key
modelstrkugel-3.5TTS model (kugel-3.5 or kugel-3), see Models
voice_idintrequiredVoice ID to use for synthesis
sample_rateint24000Output sample rate in Hz
cfg_scalefloat2.0CFG scale, clamped to [1.2, 2.5]
max_new_tokensint2048Maximum tokens to generate (1 to 2048)
languagestr | NoneNoneISO 639-1 language code (e.g., en, de). Unset: recognized per sentence, falling back to the voice's primary language
normalizeboolTrueApply text normalization
project_idint | NoneNoneProject whose pronunciation dictionaries apply. Unset: no dictionary applies
dictionary_idslist[int] | NoneNoneDictionaries by ID; [] disables them. A non-empty list requires project_id
regionstr | NoneNoneSet "eu" for the direct EU endpoint; overridden by base_url
base_urlstrhttps://api.kugelaudio.comAPI base URL
on_word_timestampsCallable[[str, list[WordTimestamp]], None] | NoneNoneCallback receiving the context ID and each batch of word timestamps. Setting it turns word timestamps on, see Word timestamps

STT service parameters

Keyword arguments of KugelAudioSTTService(...). Every one is optional; one left unset is not sent, so the server default applies.

ParameterTypeDefaultDescription
api_keystrKUGELAUDIO_API_KEY envYour KugelAudio API key. An eu- prefix selects the EU endpoint
languagestr | list[str] | NoneNoneA code such as "de", a list of codes to detect among, or "auto". Unset: German or English, detected
turn_detectionstr | NoneNoneWho ends a turn: "kugelturn" (server default), "vad" (a fixed pause) or "manual"
turn_detection_levelstr | NoneNoneWith "kugelturn": "responsive" (server default) or "conservative"
boosted_phraseslist[str] | NoneNoneNames, brands and domain terms to spell as written
project_idint | NoneNoneProject whose dictionaries apply. Unset: the API key's project
dictionary_idslist[int] | NoneNoneDictionaries to apply; [] applies none. Unset: every active dictionary of the project
modelstr | NoneNoneSpeech-to-text model ID. Unset: the API's default model
sample_rateint16000Rate in Hz the audio is sent at; pipeline audio at another rate is resampled to it
regionstr | NoneNoneSet "eu" for the direct EU endpoint; overridden by base_url
base_urlstrhttps://api.kugelaudio.comAPI base URL

Supported TTS sample rates

RateNotes
24000Native rate (recommended)
22050CD quality
16000Wideband telephony
8000Narrowband telephony

Lower rates are resampled server-side. Any other value raises ValueError in the constructor.

Performance

Set language when you know it (Text normalization), and call prewarm() inside your async setup code (it needs a running event loop and does nothing otherwise) so the first reply does not pay the WebSocket handshake. The service then reuses that connection across run_tts() calls and reconnects transparently if it drops. On Pipecat 1.x each assistant turn still opens a new server-side context; the service creates it when the LLM starts responding, so the setup overlaps the LLM's time-to-first-token. The KugelAudio TTFA: log line measures text send to first audio chunk on the WebSocket, without LLM or STT time. See Latency for how to measure end to end and Chunking & per-segment latency for how the server chunks text.

Usage Patterns

Updating the Voice at Runtime

Python
# Switch voice mid-conversation (closes the cached WebSocket first)
await tts.set_voice("300")

Pipeline Frame Flow

The KugelAudioTTSService emits standard Pipecat frames, in this order:

  1. TTSStartedFrame: audio generation has begun
  2. TTSAudioRawFrame: raw PCM audio chunks (16-bit, mono), one or more
  3. TTSStoppedFrame: audio generation is complete
  4. ErrorFrame: if an error occurs during synthesis

Metrics Support

The service reports Pipecat's TTFB metric (Pipecat's name for time-to-first-audio, measured from request to first audio chunk) and TTS usage (characters per request) automatically; tts.can_generate_metrics() returns True.

Removing Background Noise

KugelAudioEnhanceFilter puts KugelAudio speech enhancement (Clarity, clarity-1) on the transport's audio input, the audio_in_filter slot Pipecat uses for noise filters. It removes background noise before VAD and STT hear the user. Pass speaker= with a clean 2 to 8 second sample of one person and it keeps only that voice.

The Quick Start bot with enhancement on the microphone input. Save it as bot.py, set the same two API keys, and run python bot.py:

Python
import asyncio
import os

from pipecat.frames.frames import TTSSpeakFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.local.audio import (
    LocalAudioTransport,
    LocalAudioTransportParams,
)
from pipecat.turns.user_turn_strategies import ExternalUserTurnStrategies

from kugelaudio import load_audio
from kugelaudio.pipecat import (
    KugelAudioEnhanceFilter,
    KugelAudioSTTService,
    KugelAudioTTSService,
)


async def main():
    transport = LocalAudioTransport(
        LocalAudioTransportParams(
            audio_in_enabled=True,
            audio_in_filter=KugelAudioEnhanceFilter(),  # reads KUGELAUDIO_API_KEY
            # keep one voice: KugelAudioEnhanceFilter(speaker=load_audio("caller.wav"))
            audio_out_enabled=True,
            audio_out_sample_rate=24000,
        )
    )
    # The server hears when the user has finished; no local VAD needed.
    stt = KugelAudioSTTService(language="en")
    llm = OpenAILLMService(
        api_key=os.environ["OPENAI_API_KEY"],
        settings=OpenAILLMService.Settings(
            model="gpt-4o-mini",
            system_instruction="You are a helpful voice assistant. Keep answers short.",
        ),
    )
    tts = KugelAudioTTSService(
        model="kugel-3.5", voice_id=1071, sample_rate=24000, language="en"
    )
    tts.prewarm()

    context_aggregator = LLMContextAggregatorPair(
        LLMContext(),
        # Let the STT's turns drive the conversation.
        user_params=LLMUserAggregatorParams(
            user_turn_strategies=ExternalUserTurnStrategies()
        ),
    )
    pipeline = Pipeline(
        [
            transport.input(),
            stt,
            context_aggregator.user(),
            llm,
            tts,
            transport.output(),
            context_aggregator.assistant(),
        ]
    )

    task = PipelineTask(pipeline)
    await task.queue_frames([TTSSpeakFrame("Hi! How can I help you?")])
    await PipelineRunner().run(task)


if __name__ == "__main__":
    asyncio.run(main())

Any Pipecat transport takes the filter the same way, as audio_in_filter= in its TransportParams. Push FilterEnableFrame(False) to switch it off mid-call (the stream closes and audio passes through) and FilterEnableFrame(True) to switch it back on.

Before you turn it on:

  • Latency. Enhancement runs on KugelAudio's servers, not on the device, so the audio makes a round trip over the network and the bot hears the user somewhat later than with an on-device filter. Frames keep their size; they arrive delayed.
  • Plan and limits. Enhancement uses your organization's Clarity plan: the free month, then paid. Every second of input audio is enhanced, silence included, and each running pipeline holds one stream open, so on the free month its concurrency limit caps how many sessions can be enhanced at once. See Free month and paid plan.
  • Connection. The filter opens its connection and readies enhancement on the server when the pipeline starts, before the first audio, and closes the connection when the pipeline stops. After 60 seconds without audio the server closes it and the next audio reconnects.
  • Failures. While the stream connects (after a quiet period enhancement can take several seconds to start; the filter waits up to 30 seconds), or reconnects after a rate limit or a network error, the bot hears the original audio and a warning is logged for a failure. A rejected API key, spent credits, an ended free month or an invalid request turn enhancement off for the rest of the session with one error log; pass on_error= to be told.
ParameterDefaultDescription
api_keyKUGELAUDIO_API_KEY envAPI key.
clientNoneAn existing KugelAudio client, instead of api_key and base_url.
model"clarity-1"Enhancement model.
speakerNoneload_audio(...) sample of the one voice to keep. Unset: remove noise, keep every voice.
max_backlog_s2.0Seconds the server may fall behind before the original audio is used and the stream reconnects.
on_errorNoneCalled once with the error that turned enhancement off.
base_urlhttps://api.kugelaudio.comAPI base URL.

Environment Variables

VariableRequiredDescription
KUGELAUDIO_API_KEYYesYour KugelAudio API key, used by the STT and the TTS
OPENAI_API_KEYYes*Required if using OpenAI LLM

Troubleshooting

The STT is refused with 403

Two causes: speech to text is not enabled for your organization yet (the message says Transcription is not enabled for this organization.), or the API key lacks the Speech to text permission. Ask us to enable it, or create a key with the permission.

Turns do not follow the STT

Before Pipecat 1.11 the user aggregator uses its own VAD and turn analyzer unless told otherwise. Pass user_turn_strategies=ExternalUserTurnStrategies() as in the Quick Start.

API key not found

Make sure KUGELAUDIO_API_KEY is set in your environment or pass api_key directly:

Python
tts = KugelAudioTTSService(api_key="your-api-key", voice_id=1071)
Unsupported sample rate error

The TTS service accepts 24000, 22050, 16000 and 8000. Set the same rate on the service and on your transport's audio output (in the Quick Start, sample_rate=24000 and audio_out_sample_rate=24000).

WebSocket connection fails

Verify your base_url is correct and the KugelAudio API is reachable. The service connects via WebSocket (wss://) for audio streaming. If a persistent connection drops, the service automatically reconnects on the next run_tts() call.

High latency

Check Performance first: language unset and prewarm() not called (or called outside the event loop) are the usual causes. Then measure from the same region as the endpoint, see Measuring TTFA correctly.

Python version incompatibility

The Pipecat integration requires Python 3.11 or newer. Check your version:

Bash
python --version

Next Steps