Skip to content
KugelAudioDocs

Word timestamps

Receive word-level timestamps alongside streaming KugelAudio audio for subtitles, lip-sync and barge-in handling, with the payload format and where they apply.

On this page

When word_timestamps: true is set, the server performs forced alignment on each generated audio chunk and sends a word_timestamps message shortly after the corresponding audio. Useful for barge-in handling ("which word was the agent on when the user interrupted?"), subtitle synchronization, and lip-sync.

Streaming with word timestamps

Python
for chunk in client.tts.stream(
    text="Hello, this is streaming with timestamps.",
    model_id="kugel-3.5",
    voice_id=1071,
    word_timestamps=True,
):
    if hasattr(chunk, 'audio'):
        play_audio(chunk.audio)
    elif isinstance(chunk, list):
        # Word timestamps arrive as a list of WordTimestamp objects
        for ts in chunk:
            print(f"{ts.word}: {ts.start_ms}-{ts.end_ms}ms")
            # Your own words this one speaks (None on an older server)
            if ts.original_char_start is not None:
                print("  yours:", ts.original_text[ts.original_char_start:ts.original_char_end])

The timestamp payload

Each word_timestamps message carries the alignments for one audio chunk. For the input "About 12,000 visitors.":

JSON
{
  "word_timestamps": [
    {"word": "About", "start_ms": 0, "end_ms": 290, "char_start": 0, "char_end": 5, "original_char_start": 0, "original_char_end": 5, "score": 1.0},
    {"word": "twelve", "start_ms": 310, "end_ms": 640, "char_start": 6, "char_end": 12, "original_char_start": 6, "original_char_end": 12, "score": 1.0},
    {"word": "thousand", "start_ms": 650, "end_ms": 1080, "char_start": 13, "char_end": 21, "original_char_start": 6, "original_char_end": 12, "score": 1.0},
    {"word": "visitors.", "start_ms": 1100, "end_ms": 1630, "char_start": 22, "char_end": 31, "original_char_start": 13, "original_char_end": 22, "score": 1.0}
  ],
  "chunk_id": 0,
  "text": "About twelve thousand visitors.",
  "original_text": "About 12,000 visitors."
}
FieldTypeDescription
wordstringThe aligned word, as spoken
start_msintStart time in milliseconds (relative to chunk start)
end_msintEnd time in milliseconds (relative to chunk start)
char_startintStart offset of word in the frame's text, counted in Unicode code points
char_endintExclusive end offset of word in the frame's text
original_char_startintStart offset in the frame's original_text of the words this word speaks
original_char_endintExclusive end offset in the frame's original_text
scorefloatCompatibility field; currently always 1.0

The frame itself carries:

FieldTypeDescription
chunk_idintThe chunk the timestamps belong to, as in generation_started and the audio frames
textstringThe text the chunk spoke, after normalization and pronunciation dictionaries
original_textstringThe part of your text the chunk spoke, before normalization. On /ws/tts it is the whole request text

Timestamps are relative to the start of their chunk: to place words on a global timeline, accumulate the duration of previous chunks.

In the SDKs every word carries original_char_start, original_char_end and its frame's text and original_text (originalCharStart, originalText and so on in JavaScript and Dart, getOriginalCharStart() in Java; the fields are unset when an older server does not send them), so the offsets index the strings on the same object. The types reference lists each SDK's fields.

Aligning to your own text

Normalization rewrites some of your words (12,000 is spoken as "twelve thousand"), so the spoken words and your words do not always pair one to one. original_char_start/original_char_end point every spoken word at the words of yours it speaks:

  • Several spoken words can share one span: "twelve" and "thousand" both point at 12,000. Your word starts at the first one's start_ms and ends at the last one's end_ms.
  • A word of yours that nothing speaks (an emoji, a lone symbol) is covered by no span.
  • A word the normalizer added without replacing one of yours gets an empty span (original_char_start == original_char_end) at the place it was added.

All offsets, char_* and original_char_*, count Unicode code points: an emoji is one position, whatever your language's string type counts (UTF-8 bytes in C++ and Rust &str, UTF-16 units in JavaScript, Java and Dart). Convert before slicing, as the examples above do.

Spans cover whole whitespace-separated words, punctuation included: visitors. spans 9 characters.

On the streaming endpoints, original_text is your text as the chunk received it: <prosody> tags are removed, a sentence-final ! reads as ., and the whitespace between sentences in one chunk is a single space. To place its offsets in your full text, match the chunk's words to yours in order, the same way across chunks.

Where timestamps are available

SurfaceHow they arrive
SDK one-shot streaming (stream)onWordTimestamps callback (JS/Java) or timestamp items in the chunk iterator (Python)
SDK reusable streaming sessiononWordTimestamps callback (JS/Java), or on_word_timestamps plus last_word_timestamps (Python)
SDK generate() (Python)AudioResponse.word_timestamps: the SDK streams over WebSocket internally
/ws/tts, /ws/tts/stream, and /ws/tts/multiword_timestamps frames interleaved with audio frames (reference)
REST /v1/tts/generateNot supported: the field is rejected with 400; use a WebSocket endpoint or an SDK

LiveKit uses these alignments natively for transcript sync. See the LiveKit integration.