It Predicts the Next Token, and Almost Everything Follows From That

A large language model does one thing: given a sequence of text, it produces a probability distribution over what comes next. That is the whole mechanism.

Everything else — the apparent reasoning, the code, the summarisation, the conversation — emerges from doing that one thing extremely well, at enormous scale, and then sampling from the result repeatedly.

This is worth internalising because almost every surprising behaviour of these systems, good and bad, follows from it. Why they are fluent. Why they hallucinate. Why prompting works at all. Why they are confident when wrong. Why they can write a sonnet but miscount the letters in a word. None of these are bugs bolted onto an otherwise sound reasoning engine. They are consequences of what the thing actually is.

Tokens, not words

Text is split into tokens before the model sees it. A token is roughly a common word or a word fragment — "unhappiness" might be three tokens, a rare name might be five, and a space is usually part of the token following it.

Several practical consequences follow immediately.

The model does not see letters. Asking how many times a letter appears in a word is asking it to introspect about something it never had access to, which is why models have historically been poor at it while being excellent at tasks that appear far harder.

Costs and limits are measured in tokens, not words. English runs roughly three tokens for every four words; code and unusual languages are considerably less efficient.

And the context window — how much the model can consider at once — is a token budget. Everything competes for it: the instructions, the retrieved documents, the conversation so far, and the response.

Training happens in stages, and they do different things

Pretraining is where the capability comes from. The model is shown enormous quantities of text and trained to predict the next token. No labels, no human annotation — the text is its own supervision. To predict well across a huge variety of text, it has to acquire a great deal: grammar, facts, idioms, code syntax, patterns of argument. This stage is where the expense sits, and it is why only a few organisations do it.

Fine-tuning adapts the result to follow instructions. A pretrained model is a text continuer, not an assistant — ask it a question and it might produce more questions, because that is a plausible continuation. Fine-tuning on examples of instructions and good responses turns the continuer into something that answers.

Preference training uses human judgements about which of two responses is better to shape tone, helpfulness and refusal behaviour. This is the stage that makes a model feel like an assistant rather than a text generator, and it is largely where a model's character comes from.

The important consequence: capability comes from pretraining, behaviour comes from the later stages. A model's knowledge is fixed at pretraining and has a cutoff date, and no amount of prompting adds facts it does not have.

Hallucination is the mechanism working correctly

A model generates the most plausible continuation. Plausible is not the same as true, and the model has no separate store of facts to check against — the facts, such as they are, are distributed across billions of weights as statistical regularities.

So when asked for a citation that does not exist, it produces something shaped exactly like a real citation: plausible authors, a plausible journal, a plausible year. It is not lying, and it is not malfunctioning. It is doing precisely what it does, in a case where the most plausible continuation happens to be false.

This is why hallucination cannot simply be patched out. It is the same process that produces correct answers, applied where the training data did not constrain the output enough. Reducing it means changing what constrains the output — which is what retrieval does.

The practical rule for anyone building with these systems: a model's fluency carries no information about its accuracy. Confident phrasing is a property of the training data, not evidence. Anything verifiable should be verified, and anything consequential should be grounded in a source the system can quote.

Retrieval is the main architectural answer

Retrieval-augmented generation addresses the two biggest limitations — stale knowledge and fabrication — by supplying the facts rather than hoping the weights contain them.

The shape is straightforward. Documents are split into chunks and converted to embeddings, vectors positioned so that semantically similar text lands nearby. A question is embedded the same way, the nearest chunks are retrieved, and those chunks are placed in the prompt with an instruction to answer from them.

The model now has the facts in front of it, can quote them, and can be instructed to say when the answer is not present. Updating knowledge means updating documents rather than retraining anything.

Most of the difficulty in practice is not the model but the retrieval. Chunks too small lose context; too large waste the budget and dilute relevance. Pure semantic search misses exact identifiers and product codes, which is why hybrid approaches combining keyword and vector search usually outperform either. And if retrieval returns the wrong passages, a perfect model produces a confidently wrong answer grounded in irrelevant text.

Prompting, and why it works

A prompt is not a command. It is the beginning of a sequence the model continues, and effective prompting is mostly arranging for the correct continuation to be the most plausible one.

A few things reliably help. Being specific about the desired output — format, length, audience — because vague instructions leave the model to pick from many plausible shapes. Providing examples, which is remarkably effective because it demonstrates the pattern rather than describing it. Asking for reasoning before the answer on multi-step problems, since the model's computation happens as it generates, and tokens spent reasoning genuinely improve the result. And giving it an acceptable way out — permission to say it does not know — because without one the most plausible continuation to a question is always an answer.

What does not reliably help is politeness, urgency, or elaborate persona instructions beyond setting a register. These shape tone more than quality.

Temperature and sampling

The model outputs probabilities; something has to choose. Temperature controls how that choice is made. Near zero, the most likely token is taken every time, giving consistent, conservative output. Higher values flatten the distribution and admit less likely tokens, giving variety and surprise.

The mapping to tasks is direct. Extraction, classification and anything requiring consistency want low temperature. Brainstorming and creative writing benefit from higher. A system producing structured output at high temperature is a system that will occasionally produce malformed output for no reason anybody can reproduce.

Context windows are large, and that is not the same as useful

Context windows have grown enormously, and it is tempting to treat them as a replacement for retrieval — put everything in and let the model sort it out.

Two things argue against it. Cost and latency scale with what you send, on every single request. And models attend unevenly across a long context: material at the beginning and end is used more reliably than material in the middle. A critical instruction buried in the middle of a very long prompt may effectively be ignored.

A large window is best thought of as headroom for genuinely long documents, not as a reason to stop being selective.

Evaluation is the hard part, and it is underestimated

Traditional machine learning has a test set and a metric. Generative output has neither in any straightforward sense: there is no single correct answer, and quality is multi-dimensional.

What works in practice is less elegant than people hope. Build a set of real cases with known good answers and check against them on every change — this is the unglamorous foundation and it is what most teams skip. Use another model as a judge for subjective qualities, while remembering that the judge has its own biases, including a preference for longer answers. Measure the things that are objectively checkable: does the output parse, does it cite a real source, does the code run.

And have humans look at a sample regularly. Automated evaluation drifts away from what users care about, and the only way to notice is to read the output.

The risks worth taking seriously

Prompt injection is the one that most needs understanding. If your system puts untrusted text into the prompt — a web page, an email, a document — that text can contain instructions, and the model has no reliable way to distinguish data from instruction. Treat any content that reaches the model from outside as potentially adversarial, and never let model output trigger a consequential action without a check.

Data leakage in the other direction: whatever is sent in a prompt goes to the provider, and sending confidential data into a third-party service is a decision that should be made deliberately.

Automation bias is the quiet one. People trust fluent output more than they should, and a system that is right ninety-five per cent of the time trains its users to stop checking — which is exactly when the remaining five per cent does damage.

Where this is genuinely useful

The honest assessment is that these systems excel at transformation and struggle with ground truth.

Strong: summarising, rewriting, translating between formats, drafting, explaining, extracting structure from unstructured text, writing code with a human reviewing it, and anything where a knowledgeable person checks the output. In all of these the model is manipulating text that was supplied rather than recalling facts.

Weak: anything requiring guaranteed accuracy without verification, precise arithmetic, current events after the training cutoff, and decisions where an auditable explanation is required.

The pattern worth noticing is that the successful applications nearly all keep a human in the loop or ground the output in retrieved sources. The ones that disappoint tend to be those that treated the model as a database.

Structured output, and getting something a program can use

A great deal of practical work needs the model to produce something another

system will consume — JSON, a classification label, a set of extracted fields.

Free text that usually looks right is not good enough when a parser is next in

line.

Several things help, in roughly ascending order of reliability. Describing the

required shape precisely and giving an example of it. Providing a schema.

Using a provider feature that constrains generation so that only tokens valid

for the schema can be produced, which moves the guarantee from persuasion to

mechanism. And validating the result regardless, with a retry on failure,

because a system that assumes well-formed output will eventually meet the case

where it is not.

Lowering the temperature matters here too. Structured output is precisely the

case where variety has no value and consistency is the entire requirement.

Cost, latency, and choosing a model size

Larger models are more capable and cost more per token, and both figures

matter once a system has real traffic.

The pattern that works well is to route by difficulty rather than sending

everything to the largest model available. Many requests in a typical

application are straightforward — a classification, a short extraction, a

routine rewrite — and a smaller, faster, cheaper model handles them at

indistinguishable quality. Reserving the expensive model for the requests that

genuinely need it often reduces cost substantially without any visible change.

Latency deserves separate thought because it is experienced rather than

measured on an invoice. Generation is sequential: each token depends on the

one before, so a long response takes proportionally long no matter how much

hardware is available. Streaming the response as it is produced does not make

it faster but changes the experience completely, because the user sees progress

immediately rather than waiting for a complete answer.

Caching is the other lever worth using. Many applications send the same long

system prompt with every request, and providers increasingly allow that prefix

to be cached, which cuts both cost and time to first token.

Fine-tuning, and when it is the wrong answer

Fine-tuning adjusts a model's weights on your own examples, and it is reached

for far more often than it should be.

It is genuinely good at teaching a style, a format, or a narrow task shape —

making output consistently match a house tone, or reliably produce a particular

structure. It changes how the model behaves.

It is poor at teaching facts. Knowledge fine-tuned into weights is diffuse,

hard to update, impossible to cite, and no more reliable than anything else the

model half-remembers. If the requirement is that the system knows your

documentation, retrieval is the correct architecture and fine-tuning is an

expensive way to get a worse result.

The order worth following is to exhaust prompting first, then retrieval, and

only then consider fine-tuning — and when you do, to have the evaluation set

built beforehand, because without it there is no way to tell whether the

tuned model is better or merely different.

What to practise

1. Ask a model something you know deeply and count the errors. Calibrating your trust against your own expertise is the most useful hour available.
2. Build a small retrieval system over documents you know, then ask questions whose answers are not in them and see what happens.
3. Run the same prompt at temperature 0 and 1 several times each, and watch the variance.
4. Try to injection-attack your own application by putting instructions in a document it ingests.
5. Write twenty test cases with expected answers before shipping anything, and run them on every prompt change.

Generative AI rewards people who understand what the system is rather than what it appears to be. It is a very good model of how text usually continues. That turns out to be enormously useful and occasionally misleading, and knowing which situation you are in is the entire skill.