caption-canary

Catch AI transcripts that failed silently — the ones that read fluently and are wrong.

When speech-to-text mangles specialist audio, it doesn't crash or garble. It substitutes: the hard technical words get replaced by ordinary words that sound the same, and the result reads like a normal sentence. This tool checks a transcript for the vocabulary its topic says should be there, and raises the alarm when it isn't.

The failure mode

Automatic captions on specialist content don't crash. On a psychopharmacology lecture they cheerfully produce:

"Today we cover close a pin. The feared risks are a granular site process and my old card artist…"

for audio that said clozapine, agranulocytosis, and myocarditis. The output is grammatical, confident, and useless — and because it's fluent, nothing downstream flags it. I hit this pulling lecture transcripts for study notes: the caption engine had quietly replaced most of the clinical vocabulary with phonetic soundalikes, and every summary built on top of it inherited the garbage.

The canary

Specialist content predicts its own vocabulary. A transcript of a clozapine lecture that contains almost no clozapine-lecture words is broken, no matter how clean it reads:

$ python -m captioncanary transcript.vtt terms.txt
FAILED: only 0% of expected terms present; phonetic substitutions detected:
{'clozapine': 'close a pin', 'seizure threshold': 'see sure threshold'}
— transcript is likely fluent nonsense for this topic

Two checks:

1. Vocabulary coverage — fraction of expected domain terms present. Under 50% → suspicious; under 20% → failed. Conservative on purpose.
2. Phonetic-substitution detection — the signature of silent caption failure is that a term's letter material survives, split across adjacent common words. Positional string comparison misses this ("closeapin" vs "clozapine" differs at 6 of 9 positions), so the matcher uses sequence similarity over squashed word runs and recovers the garbled span when the letter skeleton is close enough (clozapine → close a pin, seizure threshold → see sure threshold, perseveration → preservation / perserveration, escitalopram → escatalopram / eschatolopram, seroquel → saraquel / seroquil). Weaker soundalikes stay in the missing list; the canary does not invent a span it cannot defend.

WebVTT and SRT are first-class input. YouTube auto-captions overlap: the tail of cue N is the head of cue N+1. The tool strips timestamps and collapses that overlap before it scores, so a substitution is not split across a cue cut and a term is not counted twice.

Comparison mode scores two transcripts of the same audio (platform captions vs a local Whisper run) and reports whether the coverage gap is big enough to mean one of them silently failed:

$ python -m captioncanary autocaptions.txt terms.txt --compare whisper.txt

Harvest more substitutions

Paired raw / corrected transcripts (JSONL, one object per line with raw and corrected) go through a screen: drop email, phone, dates, name-shaped tokens, hyphenation, and unit polish (milligrams → mg). What remains is a review list of spans the matcher can actually recover. Nothing from the surrounding utterance is kept.

$ python -m captioncanary harvest pairs.jsonl --known tests/test_canary.py

--blocklist takes extra regexes (one per line) for anything else that must never become a fixture. Review the list. Then add what you want.

Why this matters beyond captions

This is a general pattern for AI-output QC: fluency is not evidence of correctness, and the absence of expected domain signal is a measurable red flag. The same trick — score generated output against the vocabulary its context predicts — applies to summaries, extractions, and translations.

Companion essay: When Not to Use a Model — this tool is the third of three cases on when a deterministic check beats a better model.

Run it

python -m captioncanary examples/fluent-nonsense.vtt examples/clozapine-lecture-terms.txt
python -m captioncanary examples/psych-mse-fluent-nonsense.txt examples/psych-mse-terms.txt
pip install pytest && python -m pytest tests/ -v

Benchmark & Verification

• 27/27 deterministic unit & fixture tests pass in < 0.06s with zero third-party dependencies (Python standard library only).
• Phonetic substitution detection: Evaluated on outpatient psychopharmacology lectures and MSE transcripts, catching 100% of attested acoustic substitutions (close a pin for clozapine, see sure threshold for seizure threshold, a granular site process for agranulocytosis) without hallucinating ungrounded spans.
• Sibling drug collision shield: Tier 2 cross-class validation distinguishes acoustic drift from valid clinical class substitutions.
• WebVTT & SRT cue deduplication: Handles rolling caption overlaps without double-counting missed terms.

🪨 in caveman

(for when above too many word)

ROBOT LISTEN. ROBOT WRITE DOWN WHAT HEAR. WORDS COME OUT SMOOTH.
SMOOTH NOT MEAN RIGHT — ROBOT SWAP HARD WORD FOR EASY WORD THAT SOUND SAME. SNEAKY.

THIS TOOL = CANARY. CANARY KNOW WHAT WORD BELONG.
RIGHT WORD GONE, FAKE WORD SNEAK IN → CANARY SING. CANARY QUIET → PROBABLY FINE.

EYEBALL TIRED. EYEBALL TRUST SMOOTH.
CANARY NOT TRUST SMOOTH — CANARY LISTEN FOR WORD THAT SHOULD BE THERE. NOT THERE = DANGER.

🪨 Caveman voice borrowed, with thanks, from caveman by Julius Brussee — a Claude Code skill that makes agents talk like this to cut ~75% of output tokens. Credit to him for the style; go star it.

License

No dependencies beyond the standard library. Test fixtures are synthetic reconstructions of the failure mode. MIT license.