Speech Recognition

Sound gets cut into very short slices

What a microphone picks up is a vibration in the air. That vibration turns into a string of numbers and enters the computer. But ask "what was said" about that whole string at once, and no answer comes out — speech is sound that keeps changing from one moment to the next.

So it gets cut first. A second gets split into roughly a hundred slices, and each slice gets read for which pitches of sound are mixed in and how much. Laid out this way, an "ah" sound and an "sss" sound show up as clearly different shapes.

There's a reason the slices are cut so fine. A speech sound changes shape depending on the sounds around it — the same "t" sound comes out differently depending on what follows it. Cutting things finely keeps this shifting from slipping through unnoticed.

Each slice gets a score

For each slice it has cut, the model doesn't commit to one fixed answer. Instead it leaves behind a score for every candidate — something like "this slice is probably this sound, but there's some chance of that other one too."

Leaving the score behind turns out to be far more useful. Settle on one answer too hastily from a single slice, and if it doesn't fit with the sound that follows, there's no way back. Keeping several branches open and picking the most plausible path across the whole sentence works far better.

The flow around it picks the answer

Sound scores alone leave a huge number of words undecided. "Recognize speech" and "wreck a nice beach" sound almost identical, and "the flowers in the garden" and "add a cup of flour" contain a word that sounds exactly the same either way.

That's why speech recognition also carries a part that tracks the flow of the sentence. If the words before it were about a garden, that word leans toward the bloom; if they were about a recipe, it leans toward the baking ingredient. It's the same instinct a person uses to follow a conversation in a noisy room.

Because this part exists, speech recognition can reconstruct a sentence even when the sound itself came through a little blurred. On the flip side, in speech with no flow to lean on — an unfamiliar person's name, or a random string of digits — there's no flow to help, and errors climb noticeably.

These days it's learned all in one piece

It used to be built from separate parts bolted together — one part to tell sounds apart, a dictionary of how each word is pronounced, and a part that knew the flow of a sentence, each built on its own and then stitched together. Every part needed a person's hand, which made adding a new language a major undertaking.

Now, a recording and its transcript get paired up in enormous numbers, and one single model learns to go straight from sound to text as a whole. Nobody has to build the pronunciation dictionary by hand anymore, and one model can handle multiple languages and noisy recordings together.

Text appears before the sentence is even finished

When text lands on screen in real time, the system is producing an answer without knowing what's still to come. That's why text already on screen sometimes quietly changes once the next words arrive — it isn't a mistake, it's the system revising its answer once it sees more of the flow.

Spacing and punctuation aren't in the sound at all. They get added separately, based on cues like whether the pitch rises at the end or where a breath was taken. That's exactly why commas and periods often land in odd places in a transcript.