Interviews on generative AI reward people who understand what the system is rather than what it appears to be. Almost every surprising behaviour - the fluency, the hallucination, the confidence when wrong - follows from predicting the next token.
Expect questions on retrieval, because that is the main architectural answer to the model's limitations, and on evaluation, because it is the part most teams skip and the part that decides whether anything improves.
Answer each one out loud before you open it. An answer you have read is not an answer you can give under pressure, which is the only kind that counts.
Q1 What does a large language model actually do?
It produces a probability distribution over the next token, given the text so far.
Everything else - the apparent reasoning, the code, the conversation - emerges from doing that repeatedly and sampling from the result.
What it is really testing. Whether you can state the mechanism plainly rather than anthropomorphising.
Q2 What is a token?
A common word or word fragment, which is what the model actually sees instead of characters.
English runs roughly three tokens per four words. Code and unusual languages are considerably less efficient.
What it is really testing. Whether you know the model does not see words or characters.
Q3 Why are models bad at counting letters in a word?
Because they never see letters - the word arrived as one or two tokens.
It is asking the model to introspect about something it did not have access to, which is why it fails at a task that looks trivial.
What it is really testing. Whether you can explain the failure rather than calling it a limitation.
Q4 What are the stages of training a chat model?
Pretraining on vast text, fine-tuning on instructions, then preference training.
Capability comes from pretraining; behaviour comes from the later stages. That distinction explains what prompting can and cannot change.
What it is really testing. Whether you can separate capability from behaviour.
Q5 What is pretraining doing?
Predicting the next token over enormous quantities of text, with no labels required.
To predict well the model must acquire grammar, facts, idioms, code syntax and patterns of argument. This is where the expense sits.
What it is really testing. Whether you know it needs no labels, which is why it scales.
Q6 Why does a pretrained model need instruction tuning?
Because a raw model continues text rather than answering - ask a question and it may produce more questions.
Fine-tuning on instruction-response pairs turns a continuer into something that responds, which is a behavioural change rather than a capability one.
What it is really testing. Whether you know what a raw pretrained model actually does.
Q7 What is RLHF or preference training for?
Shaping tone, helpfulness and refusal behaviour using human judgements between responses.
It is largely where a model's character comes from, and why two models trained on similar data can feel very different.
What it is really testing. Whether you connect it to a model's character rather than its knowledge.
Q8 Why do models hallucinate?
Because they generate the most plausible continuation, and plausible is not the same as true.
There is no separate store of facts to check against. A fabricated citation is shaped exactly like a real one because that is what is plausible.
What it is really testing. Whether you can explain it as the mechanism working rather than failing.
Q9 Can hallucination be eliminated by better training?
Not entirely - it is the same process that produces correct answers, applied where training did not constrain the output.
Reducing it means changing what constrains the output, which is what retrieval does.
What it is really testing. Whether you know it cannot simply be patched out.
Q10 What does fluency tell you about accuracy?
Nothing. Confident phrasing is a property of the training data.
That is the single most important calibration point for anyone building with these systems or relying on their output.
What it is really testing. Whether you are calibrated. This is the most consequential point in the subject.
Q11 What is retrieval-augmented generation?
Fetching relevant documents and placing them in the prompt so the model answers from them.
It addresses stale knowledge and fabrication together, and knowledge is updated by changing documents rather than retraining.
What it is really testing. Whether you name both problems it addresses.
Q12 What is an embedding in this context?
A vector representing text, positioned so semantically similar passages are close together.
Retrieval works by embedding the question and finding the nearest chunks, which is why it matches meaning rather than keywords.
What it is really testing. Whether you can explain why it matches meaning rather than words.
Q13 Why do pure vector searches sometimes fail?
They miss exact identifiers, product codes and rare terms that keyword search handles well.
Hybrid retrieval combining both usually outperforms either, which is why production systems rarely use embeddings alone.
What it is really testing. Whether you know hybrid retrieval usually wins.
Q14 What makes chunking difficult?
Chunks too small lose context; too large waste budget and dilute relevance.
Splitting on semantic boundaries and overlapping slightly usually beats a fixed character count, but it must be tested on real questions.
What it is really testing. Whether you treat chunking as something to test rather than configure.
Q15 Where does most RAG difficulty actually lie?
In retrieval, not in the model.
If the wrong passages are returned, a perfect model produces a confidently wrong answer grounded in irrelevant text.
What it is really testing. Whether you locate the difficulty in retrieval rather than in the model.
Q16 What makes a prompt effective?
Specificity about the output, examples, room to reason, and permission to say it does not know.
A prompt is the beginning of a sequence the model continues, so the aim is making the correct continuation the most plausible one.
What it is really testing. Whether you mention giving the model an acceptable way out.
Q17 Why does asking for reasoning before the answer help?
Because the model's computation happens as it generates, so tokens spent reasoning genuinely improve the result.
Demanding an immediate answer on a multi-step problem removes the space in which the work would have happened.
What it is really testing. Whether you connect generation to where the computation happens.
Q18 What does temperature control?
How the next token is sampled - near zero takes the most likely, higher values admit less likely ones.
Extraction and classification want low temperature; brainstorming benefits from higher. Structured output at high temperature produces occasional malformed results.
What it is really testing. Whether you can map settings to task types.
Q19 What are top-p and top-k?
Alternative sampling constraints - top-k limits to a fixed number of candidates, top-p to a cumulative probability mass.
They interact with temperature, and adjusting several at once makes behaviour hard to reason about.
What it is really testing. Whether you know adjusting several at once makes behaviour opaque.
Q20 What is the context window and what competes for it?
The token budget for one request - instructions, retrieved documents, conversation history and the response.
Everything shares it, so a long history can silently crowd out the retrieved context you depend on.
What it is really testing. Whether you know history can crowd out retrieved context.
Q21 Is a large context window a substitute for retrieval?
No. Cost and latency scale with what you send, and attention across very long contexts is uneven.
Material in the middle of a long prompt is used less reliably than material at the beginning or end.
What it is really testing. Whether you know attention across long contexts is uneven.
Q22 What is prompt injection?
Untrusted content carrying instructions the model cannot reliably distinguish from yours.
Any web page, email or document placed in a prompt is potentially adversarial, which makes it an architectural concern rather than a prompt-wording one.
What it is really testing. Whether you treat it as architectural rather than a prompt-wording problem.
Q23 How do you defend against prompt injection?
Treat model output as untrusted, require confirmation for consequential actions, and limit tool permissions.
There is no prompt wording that reliably prevents it, so the defence has to be in what the system is allowed to do.
What it is really testing. Whether your defence is about permissions rather than instructions.
Q24 What is the risk of sending data to a model provider?
Whatever is in the prompt leaves your systems.
Sending confidential data to a third-party service is a decision to make deliberately rather than by default.
What it is really testing. Whether you treat it as a deliberate decision rather than a default.
Q25 What is automation bias and why does it matter here?
People trust fluent output more than they should and stop checking.
A system right ninety-five per cent of the time trains its users to stop verifying, which is exactly when the other five per cent causes damage.
What it is really testing. Whether you can name the failure mode in human terms.
Q26 How do you get reliably structured output?
Describe the schema, give an example, use constrained generation if available, and validate with a retry.
Constrained decoding moves the guarantee from persuasion to mechanism, which is the only step that is not best-effort.
What it is really testing. Whether you include validation and a retry rather than trusting the format.
Q27 Why lower the temperature for structured output?
Because variety has no value there and consistency is the entire requirement.
It is the clearest case where the default sampling settings are wrong for the task.
What it is really testing. Whether you know the default settings are wrong for this task.
Q28 How do you evaluate a generative system?
A set of real cases with known good answers, run on every change.
It is unglamorous and most teams skip it, which is why they cannot tell whether a prompt change helped or hurt.
What it is really testing. Whether you have an evaluation set at all. Most teams do not.
Q29 What are the limits of using a model as a judge?
It has its own biases, including a preference for longer and more confident answers.
It is useful for subjective qualities at scale and should be calibrated against human judgement rather than trusted outright.
What it is really testing. Whether you know the judge has its own biases.
Q30 What should you measure objectively?
Whether the output parses, whether cited sources exist, whether generated code runs.
These are cheap, unambiguous and catch a large share of failures without needing a judgement about quality.
What it is really testing. Whether you can name checks that need no human judgement.
Q31 When is fine-tuning the right tool?
For teaching a style, a format or a narrow task shape - behaviour rather than knowledge.
Knowledge fine-tuned into weights is diffuse, hard to update and impossible to cite, which is why retrieval is the right answer for facts.
What it is really testing. Whether you know it teaches behaviour rather than facts.
Q32 What should you try before fine-tuning?
Prompting, then retrieval.
Fine-tuning is expensive, harder to iterate on, and frequently produces a worse result than retrieval for the problem people reach for it to solve.
What it is really testing. Whether you have an ordering rather than reaching for tuning first.
Q33 What is a system prompt and how is it different?
Instructions that frame the whole conversation rather than one turn.
It is more durable than a user message but not a security boundary - it can be influenced by sufficiently adversarial input.
What it is really testing. Whether you know it is not a security boundary.
Q34 Why stream a response?
It does not make generation faster but changes the experience completely.
Generation is sequential, so a long response takes proportionally long. Showing progress immediately is what makes that acceptable.
What it is really testing. Whether you know it changes perception rather than speed.
Q35 How do you reduce cost in a production system?
Route by difficulty, cache repeated prefixes, and send only what is needed.
Many requests are routine and a smaller model handles them at indistinguishable quality, which often cuts cost substantially.
What it is really testing. Whether you route by difficulty rather than using one model for everything.
Q36 What is prompt caching?
Reusing the processed form of a repeated prefix such as a long system prompt.
It reduces both cost and time to first token, and it requires putting the stable content at the beginning.
What it is really testing. Whether you know the stable content must come first.
Q37 Why can the same prompt give different answers?
Sampling is stochastic, and even at temperature zero hardware non-determinism can vary results slightly.
Treating a single run as evidence of a prompt's quality is how teams convince themselves of improvements that are noise.
What it is really testing. Whether you know a single run is not evidence.
Q38 What are these systems genuinely good at?
Transforming text that was supplied to them - summarising, rewriting, extracting, drafting, explaining.
The successful applications nearly all keep a human in the loop or ground output in retrieved sources.
What it is really testing. Whether you notice the pattern across successful applications.
Q39 What are they weak at?
Guaranteed accuracy without verification, precise arithmetic, current events, and auditable decisions.
The disappointing applications are mostly those that treated the model as a database rather than as a text transformer.
What it is really testing. Whether you can name the category of application that disappoints.
Q40 What is the most useful mental model for working with one?
A very good model of how text usually continues.
That is enormously useful and occasionally misleading, and knowing which situation you are in is the entire skill.
What it is really testing. Whether you can state a mental model rather than listing capabilities.