A “hallucination” isn't a glitch or a lie. It's the model doing exactly what it always does — generating the most plausible next text — in a moment when plausible happens to be false.
Ask for a court case and watch it build one — piece by piece — always choosing the most likely next fragment. Every part will look exactly right.
It never opened a database of cases. To the model, a real citation and a plausible-looking one are the same kind of text — built the same way, from probabilities. This is why famous fake citations have shown up in real court filings: the tool worked exactly as designed.
A common instinct is to just add the instruction. Try it — add the line to the prompt and generate again.
The instruction only added words to the window — it didn't install a fact-checker. There's no separate part of the model that knows whether its own output is true; generating plausible text is the only thing it does. It can't flag its citation as fake because, from the inside, it looks identical to a real one. Telling it “don't hallucinate” is like telling a printer to only print true sentences.
If the answer isn't in the window, don't ask the model to recall it — retrieve it. Pull a real document into the context window and have the model answer only from that. This is called RAG — retrieval-augmented generation.
When I need something I can stand behind, I never ask the model to recall it from memory.
I lean on retrieval — live web search or a database lookup — to pull real, current sources into the window, then ask the model to work only from those, with citations I can open and check myself. That's my preferred way of working, not the only valid one. The discipline underneath it is simple, and it's yours to keep: trust, but verify — and give the model something real to read.
A hallucination is the machine doing its job in a moment when its job and the truth part ways. Knowing that, you stop asking it to remember — and start handing it real things to read.
Back to The Science of AI