NLP

Why human language is hard

The language a computer handles well is language with airtight rules — one symbol locked to one meaning, in a fixed order. Human language is the opposite. The same word means something different depending on where it sits, the obvious part gets dropped entirely, and meaning gets filled in from whatever the speaker and listener both already know.

Take a single sentence: "I saw her duck." A person figures out from the surrounding situation whether that means she owns a bird or she ducked her head, almost instantly. Looking at the sentence alone, though, there's no way to settle it. Forks like this hide inside human language, several per sentence. NLP is the work of narrowing that fork down using the surrounding context.

The stages happening inside the booth

Traditionally, the job gets split into several stages. First, a sentence gets cut into workable units. Then each unit gets tagged for whether it's a noun or a verb, and what ending is attached. Next, the relationships get traced — which word is modifying which. Finally, whatever the goal is — translation, sorting, whatever it may be — gets done using that result.

Each stage stands on top of the one before it. Cut a unit in the wrong place and everything after it comes out wrong too. These days, one large model often handles all these stages at once, but the character of what's happening inside hasn't changed much.

From rulebooks to piles of examples

Early on, a person wrote grammar rules in by hand. The trouble was that human language doesn't run on rules the way it's supposed to. Add a rule to close off one exception, and another spot would break, over and over.

What came next was statistics instead of rules. An enormous pile of text that had actually been written got gathered, and how often one word followed another got counted. The standard wasn't whether it matched the grammar — it was whether people actually wrote it that way.

Now it's gone a step further. A model that reads a massive amount of text and works out the relationships between words on its own handles several jobs at once. Instead of building a separate booth for every job, one wide booth got built, and it just takes a different order each time.

Where it shows up now

Finding a document with a similar meaning to a sentence typed into a search box, shrinking a long document down to a few lines, automatically sorting whether a review is praise or a complaint — all of that is NLP. Splitting customer inquiries by type and routing them to the right person, and fixing spelling, both belong here too.

Even in something like a voice assistant, once the sound has been turned into text, this branch takes over from there. Figuring out what was asked and connecting it to the right function is the part this booth handles.

Where English makes it harder

English words carry an unusual number of senses depending on context. A word as ordinary as "set" has hundreds of dictionary definitions, and picking the right one takes the whole sentence around it, not just the word itself.

Pronouns add their own tangle. "The trophy didn't fit in the suitcase because it was too big" — whether "it" points back to the trophy or the suitcase isn't settled by the grammar at all; it takes knowing something about the world, namely that a bigger thing doesn't fit inside a smaller one.

Sarcasm piles a further layer on top. "Great, another Monday" uses words that sound positive to mean the opposite, and nothing in the sentence itself marks that flip — it takes tone, or a shared sense of what's normal, that the words alone don't carry.