LSTM
A recurrent network that uses gates to choose what to keep
- LSTM (Long Short-Term Memory) is a recurrent network with a separate long-lasting memory strand attached to it.
- Three gates open and close at every step. Each decides what to erase, what to write down freshly, and what to bring out and show.
- That memory strand keeps flowing unchanged unless something touches it. So even a word from the very start of a sentence can survive all the way to the end.
- The signal used for correcting mistakes during training also flows along this same strand, which is how the network can learn from the early parts of a long passage.
- Nobody hand-sets the rule for when a gate opens or closes. What to keep is itself learned through training.
Contents
1The analogy
Picture a line of people passing a message down the line by whisper, and now hand the line a notebook too — one that travels down the row alongside the whispering. Every time a person hears the next bit of the message, they decide three things: whether to erase a line in the notebook that's no longer needed, what from what they just heard to write down freshly, and what to pull out of the notebook and whisper to the next person.
Whatever's written in the notebook stays exactly as it is unless someone touches it. So the word "today," said at the very start, arrives just as clearly at the last person in line, as long as nobody erases it. The whisper passed sideways is still short and gets garbled quickly, but anything that needs to be held onto for a long time gets carried by the notebook instead.
2In detail
Two strands flow — the notebook and the whisper
An ordinary recurrent network only hands off one summary at each step. LSTM adds one more strand on top of that. One is the whisper passed straight to the next person; the other is the notebook, flowing long down the line.
The whisper carries what's needed right now — it gets used immediately to predict the next step or produce an answer. The notebook holds what might be needed later. Even if it isn't used right away, it keeps getting carried forward.
The two strands also travel different paths. The whisper gets fully recalculated from scratch at every single step. The notebook only ever gets erased or written to — never rebuilt. That difference is the heart of LSTM.
The erase gate: dropping what's no longer needed
The first gate scans the notebook and decides what to erase. After hearing the next bit of the message, some of what got written earlier turns out to be no longer useful. If the story's main character just changed, there's no reason to keep holding onto the old one.
Erasing doesn't mean wiping the whole thing at once. For every line, it decides how much to keep, as a value between 0 and 1. Some lines get left untouched, some get faded halfway, and some get erased completely.
The write gate: keeping only what's worth adding
The second gate decides what from what was just heard is worth writing into the notebook. Write down everything heard and the notebook fills up fast, so only what might be useful later gets picked out.
This gate splits into two parts. One shapes what the new content would be; the other decides how boldly to write it in. The product of the two is what actually gets added to the notebook. Laid onto the space freed up by the erase gate, this addition is what updates the notebook for that step.
The read gate: pulling out only what's needed now
The third gate decides what to pull from the notebook and whisper to the next person. It doesn't read the whole notebook aloud. It picks out only the part that's needed right here, right now, and sends that out.
Thanks to this, the notebook gets to quietly hold onto things for the long run, while the whisper stays short and only carries what's needed at each step — a clean division of labor. Whatever comes out as the visible output has always passed through this read gate first.
Why it holds up better over long passages
In an ordinary recurrent network, the summary gets folded entirely back into the calculation at every single step. Do that dozens of times over and the earliest content doesn't survive. LSTM's notebook flows differently. As long as the erase gate doesn't open, a value from one step carries almost unchanged into the next.
The same holds during training. When the error gets traced backward from the end to the start, this strand acts as a shortcut, letting the signal reach all the way back to the beginning of the passage. Where an ordinary recurrent network struggles past a few dozen steps, LSTM can handle several hundred, and for a long stretch it was the workhorse behind translation and speech recognition.
Its limits stayed in place, though. It can still only move forward one step at a time in order, which makes it hard to batch the computation, and with three gates attached, the amount of computation at each step grows too.
3More precisely
LSTM is a recurrent network that carries a separate vector called the cell state forward at every step, with a forget gate, an input gate, and an output gate each deciding, as a value between 0 and 1, how much to let through. Because the cell state only ever gets updated through multiplication and addition, the signal used for correcting errors during training doesn't fade away easily as it travels back.
The analogy breaks down in a place too. The notebook isn't a sentence written in letters — it's a bundle of numbers, unreadable even if a person opened it up. The erase gate also doesn't pick out one particular line to erase; it decides, for every single position all at once, how much to keep. There's no clean split either, where one specific slot is simply "in charge of the main character's name."
Lighter variants that trim down the gate count are also widely used. They differ only in how many gates there are and how the strands get merged — the underlying idea, a separate strand that flows unchanged unless something touches it, stays exactly the same. It can also be freely combined with reading a sentence once forward and once backward and merging the two results.
4Try it yourself
5Common misconceptions
It's easy to think an LSTM remembers everything that came before, but actually its capacity is fixed, so it constantly has to choose what's worth keeping.
It's easy to think a person hand-writes the rule for the three gates, but actually when to open and close them is learned entirely on its own through training.
It's easy to think an LSTM is a completely different structure from a recurrent network, but actually it's the same skeleton with a long-lasting memory strand and gates attached on top.
7One-line summary
In shortAn LSTM manages a long-lasting memory strand through three gates that erase, write, and read at every step, letting a recurrent network hold onto the start of a passage even in a long one.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02