Text accuracy and timestamp accuracy need separate acceptance tests. A transcript can contain the right words while a caption editor highlights them too early, clips their beginnings, or makes an editor repair every cue. That is a workflow risk to test for—not a measured failure of Gemini 3.5 Transcribe.
Google’s August 26 announcement describes word-level timestamps and speaker attribution for pre-recorded audio through the Interactions API, using gemini-3.5-transcribe. It lists the Live API route separately as gemini-3.5-transcribe-live. Do not infer the same timestamp contract for both from the model family name.
The announcement reports word error rates, but gives no word-boundary error distribution. Its description of disfluency cleanup also raises a useful evaluation question: when output omits a filler or resolves a self-correction, which spoken span does the retained text align to? The article does not answer that at the API-schema level.
Start with the editor’s required granularity
A subtitle cue and an individual word are different units of acceptance. If the application only needs readable caption blocks, test cue boundaries and rendering. If it needs karaoke-style highlighting or word-level editing, require usable start/end boundaries for individual words; a subtitle export option alone does not establish that capability.
For a concrete format comparison, the transcription endpoint documentation lists srt and vtt alongside JSON and text, with whisper-1 as the model default. That establishes documented output choices, not equivalent alignment quality or Gemini Transcribe availability on that endpoint. Evaluating Google’s native route is a separate integration with separate access and configuration—not a model-name substitution.
A small acceptance test before choosing a route
This is a proposed protocol, not benchmark results:
- Freeze the task. Decide whether the deliverable is verbatim text, cleaned text, subtitle cues, or word highlighting. Specify tolerable boundary error from the editor’s UX requirements before comparing outputs; there is no universal threshold supplied by these sources.
- Build a rights-cleared test set. Include short clips with pauses, fast speech, fillers, self-corrections, background noise, and speaker changes. Include a longer recording to check whether boundary errors grow toward the end. Keep language and audio conditions recorded rather than averaging everything together.
- Annotate the reference. Mark audible word starts and ends manually and agree how to handle ambiguous boundaries. Keep a verbatim reference separate from any cleaned-text target. Unmatched, deleted, or rewritten words must be counted explicitly, not silently removed from the alignment denominator.
- Score timing separately from text. For matched words, compute absolute start and end errors against the reference. Report median and p95 for each, plus the share within the chosen tolerance, matched-word coverage, and missing/invalid intervals. Inspect speaker transitions and long pauses separately. A low error among a small matched subset is not a pass.
- Import into the real editor. Check cue segmentation or highlighting, then record hands-on repair minutes per audio minute and end-to-end completion time. A syntactically valid response is only the first gate.
Compare cost per usable output, not just per request
For each route, estimate:
cost per accepted audio minute = (API charges + retry charges + alignment/post-processing cost + editing labor cost) / accepted audio minutes
Use actual billed usage and an explicitly stated labor rate. Count rejected attempts in the numerator; if nothing passes, report failure rather than a unit cost. Keep turnaround latency beside the cost figure, since the cheapest acceptable batch workflow may not meet an interactive deadline.
The decision is conditional: choose a caption-format route if its rendered cues meet the task; evaluate native word timestamps when the editor actually requires them. Neither an SRT/VTT option nor a strong text-accuracy result replaces the alignment test. No timestamp measurements, route cost comparison, or savings claim is established here.