Word timestamps
Receive word-level timestamps alongside streaming KugelAudio audio for subtitles, lip-sync and barge-in handling, with the payload format and where they apply.
On this page
When word_timestamps: true is set, the server performs forced alignment on
each generated audio chunk and sends a word_timestamps message shortly after
the corresponding audio. Useful for barge-in handling ("which word was the
agent on when the user interrupted?"), subtitle synchronization, and lip-sync.
Streaming with word timestamps
for chunk in client.tts.stream(
text="Hello, this is streaming with timestamps.",
model_id="kugel-3.5",
voice_id=1071,
word_timestamps=True,
):
if hasattr(chunk, 'audio'):
play_audio(chunk.audio)
elif isinstance(chunk, list):
# Word timestamps arrive as a list of WordTimestamp objects
for ts in chunk:
print(f"{ts.word}: {ts.start_ms}-{ts.end_ms}ms")await client.tts.stream(
{
text: 'Hello, this is streaming with timestamps.',
modelId: 'kugel-3.5',
voiceId: 1071,
wordTimestamps: true,
},
{
onChunk: (chunk) => playAudio(chunk.audio),
onWordTimestamps: (timestamps) => {
for (const ts of timestamps) {
console.log(`${ts.word}: ${ts.startMs}-${ts.endMs}ms`);
}
},
}
);client.tts().stream(
GenerateRequest.builder("Hello, this is streaming with timestamps.")
.modelId("kugel-3.5")
.voiceId(1071)
.language("en")
.wordTimestamps(true)
.build(),
new StreamCallbacks() {
@Override
public void onChunk(AudioChunk chunk) {
playAudio(chunk.getAudio());
}
@Override
public void onWordTimestamps(List<WordTimestamp> timestamps) {
for (WordTimestamp ts : timestamps) {
System.out.printf("%s: %d-%dms%n",
ts.getWord(), ts.getStartMs(), ts.getEndMs());
}
}
}
);Word timestamps are only available on the WebSocket endpoints: the REST
/v1/tts/generate endpoint does not accept word_timestamps (strict
request validation returns 400 Bad Request). Without an SDK,
connect to a WebSocket endpoint directly:
wscat -c "wss://api.kugelaudio.com/ws/tts?api_key=YOUR_API_KEY"
> {"text": "Hello, this is streaming with timestamps.", "voice_id": 1071, "model_id": "kugel-3.5", "word_timestamps": true}word_timestamps frames arrive interleaved with the audio frames. See
the /ws/tts reference and
/ws/tts/stream reference.
The timestamp payload
Each word_timestamps message carries the alignments for one audio chunk.
For the input "About 12,000 visitors.":
{
"word_timestamps": [
{"word": "About", "start_ms": 0, "end_ms": 290, "char_start": 0, "char_end": 5, "original_char_start": 0, "original_char_end": 5, "score": 1.0},
{"word": "twelve", "start_ms": 310, "end_ms": 640, "char_start": 6, "char_end": 12, "original_char_start": 6, "original_char_end": 12, "score": 1.0},
{"word": "thousand", "start_ms": 650, "end_ms": 1080, "char_start": 13, "char_end": 21, "original_char_start": 6, "original_char_end": 12, "score": 1.0},
{"word": "visitors.", "start_ms": 1100, "end_ms": 1630, "char_start": 22, "char_end": 31, "original_char_start": 13, "original_char_end": 22, "score": 1.0}
],
"chunk_id": 0,
"text": "About twelve thousand visitors.",
"original_text": "About 12,000 visitors."
}| Field | Type | Description |
|---|---|---|
word | string | The aligned word, as spoken |
start_ms | int | Start time in milliseconds (relative to chunk start) |
end_ms | int | End time in milliseconds (relative to chunk start) |
char_start | int | Start offset of word in the frame's text |
char_end | int | Exclusive end offset of word in the frame's text |
original_char_start | int | Start offset in the frame's original_text of the words this word speaks |
original_char_end | int | Exclusive end offset in the frame's original_text |
score | float | Compatibility field; currently always 1.0 |
The frame itself carries:
| Field | Type | Description |
|---|---|---|
chunk_id | int | The chunk the timestamps belong to, as in generation_started and the audio frames |
text | string | The text the chunk spoke, after normalization and pronunciation dictionaries |
original_text | string | The part of your text the chunk spoke, before normalization. On /ws/tts it is the whole request text |
Timestamps are relative to the start of their chunk: to place words on a global timeline, accumulate the duration of previous chunks.
Aligning to your own text
Normalization rewrites some of your words (12,000 is spoken as "twelve
thousand"), so the spoken words and your words do not always pair one to
one. original_char_start/original_char_end point every spoken word at the
words of yours it speaks:
- Several spoken words can share one span: "twelve" and "thousand" both point at
12,000. Your word starts at the first one'sstart_msand ends at the last one'send_ms. - A word of yours that nothing speaks (an emoji, a lone symbol) is covered by no span.
- A word the normalizer added without replacing one of yours gets an empty span (
original_char_start == original_char_end) at the place it was added.
Spans cover whole whitespace-separated words, punctuation included: visitors.
spans 9 characters.
On the streaming endpoints, original_text is your text as the chunk received
it: <prosody> tags are removed, a sentence-final ! reads as ., and the
whitespace between sentences in one chunk is a single space. To place its
offsets in your full text, match the chunk's words to yours in order, the same
way across chunks.
Where timestamps are available
| Surface | How they arrive |
|---|---|
SDK one-shot streaming (stream) | onWordTimestamps callback (JS/Java) or timestamp items in the chunk iterator (Python) |
| SDK reusable streaming session | onWordTimestamps callback (JS/Java), or on_word_timestamps plus last_word_timestamps (Python) |
SDK generate() (Python) | AudioResponse.word_timestamps: the SDK streams over WebSocket internally |
/ws/tts, /ws/tts/stream, and /ws/tts/multi | word_timestamps frames interleaved with audio frames (reference) |
REST /v1/tts/generate | Not supported: the field is rejected with 400; use a WebSocket endpoint or an SDK |
LiveKit uses these alignments natively for transcript sync. See the LiveKit integration.