Vocoder

Making sound splits into two parts

Open up a text-reading system and it's split in two inside. The front end takes text and decides which sound to make, for how long, and at what pitch, writing it down like a picture. The back end looks only at that picture and produces an actual waveform. The back end is the vocoder.

There's a reason for the split: the two jobs are very different in character. The front end has to understand the text well, since it handles content and intonation; the back end doesn't need to care about content at all — it just needs to nail the texture of the sound. Splitting them apart means each can be trained separately, and either half can be swapped out for a better one on its own.

Changing voices gets easier too — leave the schedule alone and swap only the plumbing, and the same sentence comes out in a different voice. Work with singing or instrument sound uses the same structure.

The blueprint is missing something

The picture the front end hands over states how strong each pitch band is at every moment. But it doesn't say where the sound wave actually sits at that instant — where its peaks and troughs fall. It's like a schedule that has the volume of water but nothing about the texture of the ripples.

You can't draw a waveform without that missing information. Roughly half of what a vocoder does is plausibly filling in that gap. Fill it wrong and the sound smears or picks up a metallic edge, and generating the same picture twice can even produce subtly different sound.

This gap used to be filled by fixed calculation rules — fast, but with a telltale hum. Now it's filled by a model that's listened to countless recordings and learned "a picture like this usually turns into a wave shaped like that." Reaching a point where it's hard to tell from a human voice owes a lot to this part getting better.

The waveform is extremely dense

One cell of the blueprint typically covers a few hundredths of a second. But the actual waveform is a line of tens of thousands of values packed into every second. A vocoder has to take one sparse picture and pull out a line hundreds of times denser than it.

There are broadly two ways to build it. Generating values one after another from the front is very natural but slow — if making one second of sound takes longer than one second, it's hard to use in real time. Producing the whole thing in one shot is much faster but tends to come out rough.

These days it's common to split the job between two sides working against each other: one side generates the waveform, the other tells whether it sounds like a real recording, and having them compete produces sound that's both fast and textured.

Where quality shows up

A vocoder's quality shows up in particular spots — sounds with no fixed pattern like breath or wind noise, long sustained notes in singing, and the exact moments a sound starts or cuts off. Roughness here produces a metal-scraping noise or an underwater-sounding warble.

It varies by voice too. Voice textures the model saw a lot of in training come out smooth; unfamiliar ones wobble. That's why systems handling many different people's voices either train a vocoder on a wide range of voices or feed in a voice's characteristics as a separate input.