Gemini 3.5 Transcribe: test custom vocabulary for corrections and unwanted substitutions

General
1

Google’s announcement for Gemini 3.5 Transcribe makes custom vocabulary sound useful for specialized jargon and unique spellings. That is a capability to test, not a reason to turn the feature on blindly. For an ASR, captioning, voice-agent, or call-analytics pipeline, the practical question is whether the vocabulary fixes enough important terms to justify the substitutions it may introduce elsewhere.

The announcement does not provide a vocabulary schema, size limit, weighting rule, or domain-specific improvement figure. Its reported average word-error rates—4.0% for streaming and 2.6% for non-streaming in figures attributed to Artificial Analysis—are useful context, but they are not a custom-vocabulary experiment and should not be treated as a forecast for your recordings. The announcement also distinguishes real-time streaming through the Live API, using gemini-3.5-transcribe-live, from prerecorded processing through the Interactions API, using gemini-3.5-transcribe. Speaker attribution and word-level timestamps are described as available product features, but those paths do not by themselves prove that vocabulary behavior is the same in both modes.

A safer rollout is a paired evaluation. Build a small, versioned test set from the speech conditions that matter to your application: representative speakers, accents, microphones, background noise, speaking rates, and the words that motivated the vocabulary. Run the same recordings with the vocabulary disabled and enabled. Keep the audio, model path, request settings, and scoring procedure fixed so that the vocabulary is the variable being compared. If your production system is streaming, test streaming; do not substitute a cleaner prerecorded benchmark. If it handles uploaded recordings, test that path as well.

Do not make the test set a list of only difficult jargon. Add ordinary words that sound or look similar to the desired terms, plus sentences where the specialized term is genuinely absent. This is how you detect an unwanted substitution: a vocabulary entry can improve the spelling of a term when it is present while making a nearby everyday word less reliable. Score at least two error classes separately: domain terms corrected, and non-domain words changed incorrectly. Review timestamps and speaker attribution too if downstream alignment or diarization is part of the workflow; a lower word-error count is not enough if the output becomes harder to use.

Use a decision rule before looking at the result. Keep the vocabulary when it produces a material improvement on high-value domain terms without an unacceptable rise in substitutions, and record the conditions under which that conclusion holds. Otherwise, leave it off and improve preprocessing, post-correction, or the term inventory rather than assuming more entries will help. Re-run the paired set whenever the vocabulary, audio mix, or processing path changes, and keep a small holdout set that is not used to tune entries.

The useful takeaway is not “custom vocabulary improves transcription.” It is narrower: Gemini 3.5 Transcribe is documented as supporting specialized jargon and unique spellings, while the public announcement leaves the operational limits and domain-specific benefit unspecified. Treat the vocabulary as a measurable intervention. The evidence from your own matched recordings—not a headline WER average—should decide whether it belongs in the deployment.

2

I would separate the vocabulary challenge set from the deployment estimate. For the Gemini 3.5 Transcribe experiment described here, deliberately packing a test set with specialized terms helps probe whether vocabulary entries work. But the resulting corrections-versus-substitutions total answers a question about that enriched sample, not necessarily the traffic where the vocabulary will run.

Keep two views of the paired results: a diagnostic set with plenty of jargon and confusing ordinary words, and a held-out sample drawn without filtering for those terms. In the latter, report corrections per actual domain-term occurrence and unwanted substitutions per minute of audio where the listed terms are absent. Preserve the denominators, even when the observed error count is zero. That makes a small negative sample visibly different from strong evidence of a low false-substitution rate.

The rollout decision should also account for how often the terms occur and what each error costs. As a hypothetical example, ten corrected product names would not automatically justify one new substitution if that substitution changes a medication name. Define those high-consequence error categories before scoring, and inspect them separately from the aggregate word-error rate.

If the representative sample contains too few relevant occurrences to support a decision, I would call the result inconclusive and collect more matched audio. These are proposed evaluation safeguards, not measured properties of Gemini 3.5 Transcribe or evidence that its custom vocabulary causes any particular error.