Made Whisper transcribe 122 hours of D&D in two languages at once
We've played the same D&D campaign since 2020, in Greek with English rules words mixed in. Transcribing the recordings taught me to distrust the quiet parts of a transcript.
My friends and I have played the same Dungeons & Dragons campaign since 2020. A session runs three or four hours, and in the first year we recorded them. Our written notes start at episode 31. Before that, the story is mostly in the recordings, 38 videos and 122 hours of us talking over each other.
In August I set out to turn them into notes. The plan fit in one line: an LLM and some speech-to-text, to transcribe the old episodes and make notes out of them. The speech-to-text part turned out to be the story.
A machine that hears one language at a time
Whisper is OpenAI’s open speech recognition model. It listens to thirty seconds of audio at a time and writes down what it hears. I ran its large version on my Mac with MLX, Apple’s machine learning library, so the whole job ran on my own machine.
We speak Greek at the table, but the rules of D&D are in English, and the two mix in almost every sentence. Someone asks “Τι σέιβ είναι?”, which means “what save is it?” Someone else answers “Ένα death save.” Of the 108,239 lines in the transcripts, 10,099 contain both alphabets.
Whisper settles on one language for each stretch of audio, and for us that’s Greek. So English words often come out in Greek letters, spelled the way they sound. “Advantage” became αντβάντετς, “saving throw” became σέιβιν θρόου, and Chronurgy, a school of wizardry, became Κρόνουρτζι. When Whisper did write English, it sometimes misheard it. A constitution saving throw came out as a “constitutional saving throw”.
Words a find-and-replace would break
The obvious fix is a list of corrections: wherever you see σέιβιν θρόου, write “saving throw”. Claude Code built that list from every transcript. It holds 401 terms in 19 groups, from dice rolls and schools of magic to creatures and conditions.
Only 59 of those terms had spellings that were safe to fix automatically, and only as whole words. “Save” can also mean rescue, and “advantage” is an ordinary English word too. A blind replacement would turn a character saving a friend into a rule of the game. Every other fix needed someone to read the sentence around it.
A transcript can’t say what it missed
A transcript looks complete even when it isn’t. It goes from one line to the next, and nothing marks the minutes in between. So I had the pipeline audit its own output.
Whisper reports how sure it is of each word and each line. A line that compresses too well is usually a loop, the same words over and over. A gap of more than ten seconds between two lines could be a pause, or something Whisper skipped.
The audit flagged 18,917 lines, about one in six, and 1,785 gaps. Then a voice detector listened to the audio inside each gap. In 1,716 of them it heard something like a voice. Together those gaps lasted 12.5 hours, a tenth of everything we had recorded.
Asking twice
I cut each of those gaps out and gave it to Whisper again. This time it also got a short prompt and our list of names, to help it spell. Of the 1,716 gaps, 1,083 came back with text and 633 came back empty.
Some of that text was Whisper at its worst. The first gap sits three minutes into our introduction video. It came back as “Τις πιο σημαντικές πραγματικές πραγματικές πραγματικές”, which means “the most important real real real”, and kept going like that. Given noise and no clear words, Whisper can repeat itself like this. Other gaps came back as the words of my prompt, repeated back to me.
Seventy minutes of the game
I wasn’t going to listen to 12.5 hours of gaps myself, so Claude Code reviewed them. For each gap it read the lines before and after, the second attempt, and how much of the gap sounded like a voice. Then it chose one of three answers: write the speech down, mark the gap as holding nothing usable, or leave it unresolved.
For 834 gaps, the second attempt held nothing usable. It was empty, noise, a loop or my own prompt. Another 623 stayed unresolved, because someone probably spoke, but neither attempt gave words worth trusting. The remaining 259 gaps gave back 414 lines, about 70 minutes of the game.
Here are three of the lines that came back:
Να ρίξουμε κι εμείς που είμαστε κοντά initiative. Ναι, να ρίξετε όλοι.
“Do those of us nearby roll initiative too? Yes, everyone roll.”
13 piercing damage και 22 piercing damage από το sneak attack. Σύνολο 35.
“13 piercing damage, and 22 piercing damage from the sneak attack. 35 in total.”
Λοιπόν, θα κάνω ένα μικρό recap τι κάναμε την προηγούμενη φορά.
“So, I’ll do a short recap of what we did last time.”
The last one is someone starting a recap of the previous session, which is exactly what I was trying to rebuild. The same review also fixed 1,434 names and rules terms that Whisper had heard but misspelled. It removed 119 lines that were loops, noise or leftovers of the prompt.
Notes that admit what they don’t know
Claude Code also wrote a story guide for each of the 29 main episodes. Each guide comes from the full transcript, checked against the recap video we made after episode 19 and against our later notes. Where a name, an identity or the order of events is uncertain, the guide says so. The campaign reference tags its claims too, as confirmed, inferred or unresolved.
The voice detector overstated the problem. It flagged 12.5 hours, and about 70 minutes of that turned out to be the game. But I would never have found those minutes by reading the transcript, which looked complete from start to finish.