How Do LLMs Actually Work? 4 Mechanisms + Code Example

The one-sentence version

An LLM is a function that takes a sequence of tokens and returns a probability for every token that could come next, applied over and over.

Everything else is detail on that sentence. Four pieces of detail matter, and the rest of this article takes them in order: how text becomes tokens, what the model actually outputs, how one token gets chosen from that output, and how attention lets earlier tokens influence the prediction.

Each stage below has runnable code attached. All output shown is real output from the accompanying file, which uses no libraries and no API key.

Stage 1: Text becomes tokens

Before a model sees your text, the text is chopped into tokens — chunks that are usually a few characters long, sitting somewhere between letters and words.

A simplified tokenizer, run on two words:

'strawberry' tokens seen by model : ['straw', 'berry'] token count : 2 actual letter count : 10 'understanding' tokens seen by model : ['under', 'stand', 'ing'] token count : 3 actual letter count : 13

Real tokenizers have vocabularies of tens of thousands of learned pieces rather than the dozen used here, but the principle is identical: common sequences become single tokens, rarer words get split into several.

This single fact explains a failure everyone has seen. Ask a model how many times the letter "r" appears in "strawberry" and it may get it wrong, not from stupidity but because it received two chunks, not ten letters. The information the question asks about was partly discarded before the model started. Grammar survives tokenisation perfectly well, which is why the same model writes flawless sentences while miscounting characters in them.

Tokens also explain context limits and pricing. A context window measured in tokens is not a word count, and English text runs roughly a bit more than one token per word, with code and non-English scripts often costing more.

Stage 2: The model outputs a distribution, not a word

This is the most commonly skipped step, and the one that clears up the most confusion: at each position, the model produces a score for every token in its vocabulary.

Those raw scores are called logits. Softmax converts them into probabilities summing to one. Only then does anything resemble a chosen word.

Here is that distribution made visible. The demo counts word pairs in a tiny corpus rather than using a neural network — the counting method is primitive, but the object it produces is exactly the object a real model produces:

after 'the': cat 33.3% ########## dog 16.7% ##### rug 16.7% ##### bone 16.7% ##### mat 8.3% ## fish 8.3% ## after 'sat': on 100.0% ##############################

Two things worth noticing. After "the", the model is genuinely uncertain and spreads its belief across six options. After "sat", the context forces one answer. A real model does this across a vocabulary of tens of thousands of tokens, with the probabilities computed by a trained network rather than by counting — but the shape of the output is the same.

The real difference between this toy and GPT-class models is what the probability can depend on. Counting only looks at the previous word. A transformer's prediction can depend on any earlier token in the context, which is stage 4.

Stage 3: Sampling picks the token

The model produces probabilities. Something else has to choose one, and that something is the sampler.

This is where the most common misconception lives. When you ask the same question twice and get different answers, the model did not change its mind or learn anything. It produced a similar distribution both times, and a random draw landed differently.

Temperature controls how that draw is shaped. Low temperature sharpens the distribution toward whatever was already most likely; high temperature flattens it, giving unlikely tokens a real chance:

temp 0.2: cat=91%, dog=3%, rug=3% temp 1.0: cat=33%, dog=17%, rug=17% temp 2.0: cat=24%, dog=17%, rug=17%

The same underlying prediction, reshaped three ways. At 0.2 the model will say "cat" nearly every time. At 2.0 it is close to picking among the options at random.

Generating from the same starting word at two temperatures:

temp 0.2 sample: the cat sat on the cat sat on the temp 1.0 sample: the cat sat on the fish the cat sat

Low temperature is repetitive; higher temperature is varied and less reliable. That trade-off is the whole design space of sampling. Setting temperature to zero makes generation deterministic by always taking the highest-probability token, which is why factual and code tasks usually run at a low temperature, and creative writing does not.

The practical takeaway: variability is a setting, not a property of intelligence. If your application needs reproducibility, that is a parameter you control.

Stage 4: Attention decides what context counts

Attention is how a token's prediction comes to depend on specific earlier tokens rather than on all of them equally.

Every token gets a vector — a list of numbers positioning it in a high-dimensional space, learned during training so that tokens used in similar ways sit near each other. Attention then compares each token's vector against every earlier token's vector, and turns those similarity scores into weights that sum to one.

Computed on four tokens, "the cat that sat":

the cat that sat the 1.00 - - - cat 0.07 0.93 - - that 0.17 0.56 0.27 - sat 0.08 0.12 0.09 0.71

Read the row for "that". It puts 56% of its weight on "cat" — more than on itself. Nothing in the code says "the word that refers back to a noun". The vectors for "cat" and "that" point in a similar direction, the dot product between them is large, and the weight follows. In a real model those vectors are learned from data rather than written by hand, but the arithmetic is the one shown here.

Two structural details are visible in that matrix. The upper-right triangle is empty: this is the causal mask, which prevents a token from seeing tokens that come after it. Without it the model could cheat during training by reading the answer. And every row sums to 1, because attention distributes a fixed budget of focus rather than adding more.

Real models run many attention heads in parallel, each learning a different kind of relationship, stacked across dozens of layers, with a feed-forward network after each attention step. The count is enormous. The operation is the one above.

How the model learns all this

Training happens in stages, and the differences between them explain a lot about model behaviour.

Pretraining is next-token prediction at scale. Take an enormous corpus, hide the next token, ask the model to predict it, measure how wrong the prediction was, and adjust the parameters slightly. Repeat across a vast number of examples. Because the correct next token is already in the text, no human labelling is needed — the data supervises itself, which is why this scales to internet-sized corpora.

An efficiency detail worth knowing for interviews: during training the model predicts every position in a sequence in a single pass, with the causal mask ensuring each position only uses what precedes it. It does not step through one token at a time the way generation does.

Post-training then shapes a text predictor into an assistant. A raw pretrained model continues text; it does not answer questions, follow instructions or decline requests. Instruction tuning and reinforcement learning from human feedback produce those behaviours. This is why the same base model can behave very differently across products.

What the model ends up storing is worth being precise about: not a database of facts, but parameters that make certain continuations more probable in certain contexts. Facts that appeared many times become reliable. Facts that appeared once or never do not, which is the mechanism behind fabricated details.

Is an LLM just autocomplete?

Mechanically, yes. As an explanation of what it can do, that framing is misleading in an interesting way.

The honest version: the training objective genuinely is next-token prediction, the same objective as your phone keyboard's suggestion bar. The reason the outcome is different is that predicting the next token well, across a huge and varied corpus, requires the model to represent an enormous amount of structure. To predict the closing brace of a function correctly, something in the model must track the code's structure. To finish a sentence about the Mughal empire without contradiction, it must encode something about the subject.

So "just autocomplete" is accurate about the objective and wrong about the implication. Prediction is the training signal; the internal structure it forces is the interesting part.

The framing also has a limit worth stating: none of this establishes that the model understands anything, and that question is genuinely contested among researchers rather than settled in either direction. What is not contested is the mechanism, which is what this article covers.

What an LLM does not do

Being clear about the negatives prevents most practical mistakes.

• It does not choose words. It produces a distribution; the sampler chooses. Blame temperature, not the model, for inconsistency.
• It does not look anything up unless a tool or retrieval step is attached. Base generation runs from parameters alone.
• It does not plan ahead by default. Each token is predicted from what exists so far. Apparent planning emerges from the prediction being good, not from a separate plan.
• It does not know it is wrong. Confidence in phrasing is not a confidence estimate, and a fluent wrong answer costs the model exactly as much as a fluent right one.
• It does not learn from your conversation. Parameters are fixed at inference; anything remembered across turns lives in the context window or in an external store, not in the weights.
• It does not see characters. Tokens are the unit, which is why character-level tasks should be handed to code.

Run it yourself

The accompanying file how_llms_work.py contains all four demonstrations. It has no dependencies, needs no API key, and every output quoted above is its real output.

Three exercises, in increasing order of what they teach:

1. Change the corpus in demo 2 to a paragraph of your own writing and print the distribution after a common word. You will see immediately how a small dataset makes some continuations certain and others impossible — the toy version of the rare-fact problem.
2. Set temperature to 0 in demo 3 and generate several times. Identical output every time, which is the reproducibility knob most people do not realise they have.
3. Change the vectors in demo 4 so that "that" is closer to "sat" than to "cat", and watch the attention row move. That is the entire idea of learned embeddings, done by hand.

If you can explain what changed in each case, you understand the mechanism better than most people who have read about it — and considerably better than someone who has only watched a video about it.

---

Common Mistakes

---

Frequently Asked Questions

How do large language models actually work?

An LLM converts text into tokens, then for each position computes a score for every token in its vocabulary, which softmax turns into a probability distribution. A sampler selects one token from that distribution, appends it to the sequence, and the process repeats. Attention lets each prediction depend on specific earlier tokens rather than all of them equally.

What is a token in an LLM?

A token is a chunk of text, typically a few characters, that sits between a letter and a word. Common words are usually single tokens while rarer words split into several. Because models process tokens rather than characters, they handle grammar reliably but can fail at counting letters within a word.

Why does ChatGPT give different answers to the same question?

Because a sampler draws randomly from the model's probability distribution. The model produces a similar distribution each time; the random draw lands differently. This is controlled by the temperature setting, and setting temperature to zero makes generation deterministic by always selecting the highest-probability token.

What is temperature in an LLM?

Temperature reshapes the probability distribution before sampling. Low values sharpen it toward the most likely token, producing consistent, repetitive output. High values flatten it, giving unlikely tokens a real chance and producing varied but less reliable output. It is a sampling parameter, not a measure of creativity or intelligence.

How does attention work in simple terms?

Each token is represented as a vector. Attention compares a token's vector with those of all earlier tokens, and converts the similarity scores into weights that sum to one. Tokens with similar vectors receive more weight, so the prediction depends more on them. A causal mask prevents any token from attending to tokens that come after it.

Is an LLM just autocomplete?

The training objective genuinely is next-token prediction, the same objective as keyboard autocomplete. The difference is scale and what predicting well requires: to predict correctly across a huge corpus, the model must internally represent grammar, code structure, factual associations and discourse patterns. The objective is simple; the structure it forces is not.

What is the difference between pretraining and fine-tuning?

Pretraining is next-token prediction over an enormous corpus, requiring no human labels because the text supervises itself. It produces a model that continues text but does not follow instructions. Post-training stages such as instruction tuning and reinforcement learning from human feedback then shape that predictor into an assistant that answers, follows instructions and declines requests.

Do LLMs store facts like a database?

No. They store parameters that make certain continuations more probable in certain contexts. Facts repeated many times across training data become reliable, while facts appearing once or never do not, which is why models are dependable on well-known information and prone to fabricating obscure specifics.

---

Conclusion

Four ideas cover it. Text becomes tokens. The model turns a sequence of tokens into a probability distribution over what comes next. A sampler picks one, with temperature deciding how adventurous that pick is. Attention lets each prediction lean on the earlier tokens that matter to it.

Scale is what turns those four mechanisms into something that writes working code — billions of parameters, dozens of layers, many attention heads, an enormous corpus. But scale changes the quality of the predictions, not the nature of the operation. Run the file, print a distribution, change a vector and watch the attention row move. Ten minutes of that is worth more than another explanation, including this one.

Join Hello Engineers Free