Language Models73
How chat AIs like ChatGPT read and write. Tokens, transformers, attention, embeddings, prompts, hallucinations, RAG, context windows: the words you have probably met while using AI.
An AI that takes a goal and works through the steps on its own
AI NotebookA workspace that reads only what you put in and answers from that
AI SearchA search that reads the web for you and hands back a written answer
AttentionA way of deciding how much weight to give each input
AutoregressiveRereads what it just wrote, then adds one more piece
BERTA model trained to fill blanks by reading both sides at once
Chain of ThoughtWorking through the middle steps before landing on an answer
ChatbotA program you exchange text or speech with, back and forth
Computer-Use AgentAI that watches the screen and moves the mouse and keyboard for you
Context WindowThe most text a model can take in and look at at once
Conversation MemoryRe-reading past conversation every time to keep it going
Cosine SimilarityA ruler that measures how much two directions point the same way
Document Q&AAsk questions about one document and get answers pulled from it
EmbeddingRewriting meaning as a set of numbers that marks a spot
Encoder-DecoderA two-part design that reads with one half, writes with the other
Few-Shot LearningShowing a couple of worked examples and asking for the same
Fine-TuningNudging a trained model further with new data
Function CallingA model requesting a tool in a fixed, structured form
GPTA language-model design that only looks back and writes forward
GroundingTying each sentence of an answer to real evidence
HallucinationWhen an AI invents an answer that sounds right but is not true
Instruction TuningTeaching a model to answer in the shape a request calls for
JailbreakGetting AI to talk its way past the rules it's supposed to follow
Knowledge CutoffThe point where a model's training material stopped
Knowledge DistillationPouring a big model's judgment into a smaller one
Language ModelA program that scores what word probably comes next
Large Language ModelA language model grown enormous by reading a huge pile of text
LLM-as-a-JudgeLetting a language model rate an answer instead of a person
LogitsThe raw score each candidate gets before becoming a probability
LSTMA recurrent network that uses gates to choose what to keep
Machine TranslationHaving a machine carry writing from one language into another
Mixture of ExpertsA model structure that switches on only the branches it needs
Multi-AgentSeveral AIs splitting up roles and passing work between them
Multi-Head AttentionThe same scene watched by several viewpoints at once
MultimodalAI that handles text, images, and sound together
N-gramThe way of guessing the next word by counting the last few
Next-Token PredictionGuessing the very next piece from everything written so far
NLPThe technology that lets computers handle human language
PerplexityA score for how many directions a model is torn between
PersonaThe role and tone you assign an AI to speak as
Positional EncodingA way of tagging each spot with its place in line
PretrainingLearning the basics before anyone decides what the job will be
PromptThe written request you hand an AI, spelling out what to do
Prompt EngineeringShaping an instruction so the result matches what you want
Prompt InjectionAn attack where instructions hidden in outside material steer the AI
RAGRetrieval-Augmented GenerationFetching relevant material before answering, then writing from it
Reasoning ModelA language model that works through steps before it answers
RefusalAn AI declining to carry out a request it shouldn't fulfill
RLHFPolishing the texture of answers using what people prefer
RNNRecurrent Neural NetworkA network that carries forward a running summary, word by word
SamplingDrawing one candidate at random, weighted by how likely it is
Scaling LawThe rule for how much better performance gets as scale grows
Self-AttentionWords in the same sentence checking in with each other
Semantic SearchSearch that finds things close in meaning, not matching letters
Sentiment AnalysisSorting the tone a text shows into positive, negative, neutral
Sequence-to-SequenceListens to the end, then rebuilds the message at a different length
Small Language ModelA language model shrunk down to run on ordinary devices
SoftmaxA calculation that turns several scores into shares totaling 100%
StreamingSending each piece of an answer the moment it's ready
SummarizationPicking what to keep from a long piece and making it short
SycophancyThe habit of agreeing with the user even when they're wrong
System PromptThe operating rules a service lays down before a conversation starts
TemperatureA dial that widens or narrows the gap between candidates' odds
Text GenerationBuilding text by looking ahead and adding the next piece
TokenThe bite-size pieces an AI cuts text into before reading it
TokenizerThe device that cuts text into a fixed list of pieces and numbers them
Tool UseChoosing the right outside means and using it when needed
Top-p SamplingDrawing only from a candidate pool trimmed from the top
TransformerAn AI design where every word in a sentence looks at all the others
VectorA set of numbers for several items, written in a fixed order
Vector DatabaseA store that keeps sets of numbers so the nearest ones surface fast
Word2VecLearning a word from its neighbors and giving it coordinates
Zero-ShotGiving an instruction with no examples attached and having it work