Word timestamps
Receive word-level timestamps alongside streaming KugelAudio audio for subtitles, lip-sync and barge-in handling, with the payload format and where they apply.
On this page
When word_timestamps: true is set, the server performs forced alignment on
each generated audio chunk and sends a word_timestamps message shortly after
the corresponding audio. Useful for barge-in handling ("which word was the
agent on when the user interrupted?"), subtitle synchronization, and lip-sync.
Streaming with word timestamps
for chunk in client.tts.stream(
text="Hello, this is streaming with timestamps.",
model_id="kugel-3.5",
voice_id=1071,
word_timestamps=True,
):
if hasattr(chunk, 'audio'):
play_audio(chunk.audio)
elif isinstance(chunk, list):
# Word timestamps arrive as a list of WordTimestamp objects
for ts in chunk:
print(f"{ts.word}: {ts.start_ms}-{ts.end_ms}ms")
# Your own words this one speaks (None on an older server)
if ts.original_char_start is not None:
print(" yours:", ts.original_text[ts.original_char_start:ts.original_char_end])await client.tts.stream(
{
text: 'Hello, this is streaming with timestamps.',
modelId: 'kugel-3.5',
voiceId: 1071,
wordTimestamps: true,
},
{
onChunk: (chunk) => playAudio(chunk.audio),
onWordTimestamps: (timestamps) => {
for (const ts of timestamps) {
console.log(`${ts.word}: ${ts.startMs}-${ts.endMs}ms`);
// Your own words this one speaks (undefined on an older server)
if (ts.originalCharStart !== undefined) {
// Offsets count code points, so slice by code point, not UTF-16 unit
const points = Array.from(ts.originalText!);
console.log(' yours:', points.slice(ts.originalCharStart, ts.originalCharEnd).join(''));
}
}
},
}
);client.tts().stream(
GenerateRequest.builder("Hello, this is streaming with timestamps.")
.modelId("kugel-3.5")
.voiceId(1071)
.language("en")
.wordTimestamps(true)
.build(),
new StreamCallbacks() {
@Override
public void onChunk(AudioChunk chunk) {
playAudio(chunk.getAudio());
}
@Override
public void onWordTimestamps(List<WordTimestamp> timestamps) {
for (WordTimestamp ts : timestamps) {
System.out.printf("%s: %d-%dms%n",
ts.getWord(), ts.getStartMs(), ts.getEndMs());
// Your own words this one speaks (null on an older server)
if (ts.getOriginalCharStart() != null) {
// Offsets count code points, so convert them to String indexes
String text = ts.getOriginalText();
int from = text.offsetByCodePoints(0, ts.getOriginalCharStart());
int to = text.offsetByCodePoints(0, ts.getOriginalCharEnd());
System.out.println(" yours: " + text.substring(from, to));
}
}
}
}
);while let Some(event) = stream.next().await {
if let StreamEvent::WordTimestamps(words) = event? {
for ts in words {
println!("{}: {}-{}ms", ts.word, ts.start_ms, ts.end_ms);
// Your own words this one speaks (None on an older server)
if let (Some(start), Some(end), Some(original)) =
(ts.original_char_start, ts.original_char_end, &ts.original_text)
{
// Offsets count code points, which are Rust `char`s, not bytes
let yours: String = original
.chars().skip(start as usize).take((end - start) as usize).collect();
println!(" yours: {yours}");
}
}
}
}ka::GenerateRequest request;
request.text = "Hello, this is streaming with timestamps.";
request.voice_id = 1071;
request.word_timestamps = true;
ka::StreamCallbacks callbacks;
callbacks.on_word_timestamps = [](const std::vector<ka::WordTimestamp>& stamps) {
for (const auto& ts : stamps) {
std::cout << ts.word << ": " << ts.start_ms << "-" << ts.end_ms << "ms\n";
// Your own words this one speaks (unset on an older server)
// original_span() converts the code-point offsets to the UTF-8 substring
if (auto yours = ts.original_span()) std::cout << " yours: " << *yours << "\n";
}
};
client.tts().stream(request, callbacks);final stream = client.tts.stream(
text: 'Hello, this is streaming with timestamps.',
voiceId: 1071,
wordTimestamps: true,
onWordTimestamps: (timestamps) {
for (final ts in timestamps) {
print('${ts.word}: ${ts.startMs}-${ts.endMs}ms');
// Your own words this one speaks (null on an older server)
if (ts.originalCharStart != null) {
// Offsets count code points, so slice the runes, not UTF-16 units
final runes = ts.originalText!.runes.toList();
final yours = runes.sublist(ts.originalCharStart!, ts.originalCharEnd!);
print(' yours: ${String.fromCharCodes(yours)}');
}
}
},
);
await for (final chunk in stream) {
player.add(chunk.audio);
}Word timestamps are only available on the WebSocket endpoints: the REST
/v1/tts/generate endpoint does not accept word_timestamps (strict
request validation returns 400 Bad Request). Without an SDK,
connect to a WebSocket endpoint directly:
wscat -c "wss://api.kugelaudio.com/ws/tts?api_key=YOUR_API_KEY"
> {"text": "Hello, this is streaming with timestamps.", "voice_id": 1071, "model_id": "kugel-3.5", "word_timestamps": true}word_timestamps frames arrive interleaved with the audio frames. See
the /ws/tts reference and
/ws/tts/stream reference.
The timestamp payload
Each word_timestamps message carries the alignments for one audio chunk.
For the input "About 12,000 visitors.":
{
"word_timestamps": [
{"word": "About", "start_ms": 0, "end_ms": 290, "char_start": 0, "char_end": 5, "original_char_start": 0, "original_char_end": 5, "score": 1.0},
{"word": "twelve", "start_ms": 310, "end_ms": 640, "char_start": 6, "char_end": 12, "original_char_start": 6, "original_char_end": 12, "score": 1.0},
{"word": "thousand", "start_ms": 650, "end_ms": 1080, "char_start": 13, "char_end": 21, "original_char_start": 6, "original_char_end": 12, "score": 1.0},
{"word": "visitors.", "start_ms": 1100, "end_ms": 1630, "char_start": 22, "char_end": 31, "original_char_start": 13, "original_char_end": 22, "score": 1.0}
],
"chunk_id": 0,
"text": "About twelve thousand visitors.",
"original_text": "About 12,000 visitors."
}| Field | Type | Description |
|---|---|---|
word | string | The aligned word, as spoken |
start_ms | int | Start time in milliseconds (relative to chunk start) |
end_ms | int | End time in milliseconds (relative to chunk start) |
char_start | int | Start offset of word in the frame's text, counted in Unicode code points |
char_end | int | Exclusive end offset of word in the frame's text |
original_char_start | int | Start offset in the frame's original_text of the words this word speaks |
original_char_end | int | Exclusive end offset in the frame's original_text |
score | float | Compatibility field; currently always 1.0 |
The frame itself carries:
| Field | Type | Description |
|---|---|---|
chunk_id | int | The chunk the timestamps belong to, as in generation_started and the audio frames |
text | string | The text the chunk spoke, after normalization and pronunciation dictionaries |
original_text | string | The part of your text the chunk spoke, before normalization. On /ws/tts it is the whole request text |
Timestamps are relative to the start of their chunk: to place words on a global timeline, accumulate the duration of previous chunks.
In the SDKs every word carries original_char_start, original_char_end and
its frame's text and original_text (originalCharStart, originalText and so on in
JavaScript and Dart, getOriginalCharStart() in Java; the fields are unset when an older server does not send them), so
the offsets index the strings on the same object. The types reference
lists each SDK's fields.
Aligning to your own text
Normalization rewrites some of your words (12,000 is spoken as "twelve
thousand"), so the spoken words and your words do not always pair one to
one. original_char_start/original_char_end point every spoken word at the
words of yours it speaks:
- Several spoken words can share one span: "twelve" and "thousand" both point at
12,000. Your word starts at the first one'sstart_msand ends at the last one'send_ms. - A word of yours that nothing speaks (an emoji, a lone symbol) is covered by no span.
- A word the normalizer added without replacing one of yours gets an empty span (
original_char_start == original_char_end) at the place it was added.
All offsets, char_* and original_char_*, count Unicode code points: an emoji
is one position, whatever your language's string type counts (UTF-8 bytes in C++
and Rust &str, UTF-16 units in JavaScript, Java and Dart). Convert before
slicing, as the examples above do.
Spans cover whole whitespace-separated words, punctuation included: visitors.
spans 9 characters.
On the streaming endpoints, original_text is your text as the chunk received
it: <prosody> tags are removed, a sentence-final ! reads as ., and the
whitespace between sentences in one chunk is a single space. To place its
offsets in your full text, match the chunk's words to yours in order, the same
way across chunks.
Where timestamps are available
| Surface | How they arrive |
|---|---|
SDK one-shot streaming (stream) | onWordTimestamps callback (JS/Java) or timestamp items in the chunk iterator (Python) |
| SDK reusable streaming session | onWordTimestamps callback (JS/Java), or on_word_timestamps plus last_word_timestamps (Python) |
SDK generate() (Python) | AudioResponse.word_timestamps: the SDK streams over WebSocket internally |
/ws/tts, /ws/tts/stream, and /ws/tts/multi | word_timestamps frames interleaved with audio frames (reference) |
REST /v1/tts/generate | Not supported: the field is rejected with 400; use a WebSocket endpoint or an SDK |
LiveKit uses these alignments natively for transcript sync. See the LiveKit integration.