Your LLM System Doesn’t Have to Be
Harry Frankfurt wrote a wonderful essay called On Bullshit. His central move is a distinction most people miss: the difference between a liar and a bullshitter.
The liar cares about the truth. He has to. Truth is what a liar wants to conceal, and you cannot conceal what you have not first located. Lying is a truth-tracking activity performed in reverse.
The bullshitter doesn’t care about the truth at all. Not for it, not against it. Truth is simply not a variable in his function. His goal is to persuade, to impress, to manage how he is perceived. Whether the words happen to be true is incidental.
That’s an LLM model.
I’m not the first person to make this connection. In 2024, three philosophers at the University of Glasgow, Michael Townsen Hicks, James Humphries, and Joe Slater, published a paper in Ethics and Information Technology titled, with admirable economy, ChatGPT is bullshit. They make the Frankfurtian case rigorously. They also argue that hallucination is the wrong word for what these systems do. It implies a perceptual faculty that misfired, when the reality is that there was never a faculty there to misfire. They’re right. I’m going to keep using ‘hallucinate’ for the rest of this piece anyway, because it’s the term of art my readers actually use and I’m actually using LLM systems so that term might be more accurate (but more on that later). Hold the objection in your head while you read. It matters for what comes next.
The physics of bullshit
An LLM model takes a token, runs it through a very large pile of mathematical weights, and predicts the next token. That’s not quite right. What it actually produces is a probability distribution over next tokens. Then something samples from that distribution.
The surprising part is that if you always pick the single most probable token, you get worse results, not better. Greedy decoding produces flat, repetitive, degenerate text. So we sample instead. Temperature is the knob that controls how sharp or how flat that distribution is before we sample from it. Turn it down and the mass concentrates on the top candidates and you approach, but never reach, deterministic output. Turn it up and the mass spreads out and you get more variety, which we experience as creativity. Two related knobs, top-k and top-p, truncate the candidate pool outright.
Now read that back. Nowhere in it is there a step called check whether this is true. There is no truth register. There is no fact table it consults, no assertion it defends. There is a distribution and there is a sampler.
The model is not lying to you. Lying would require it to have a concept of truth. It is bullshitting you. It produces plausible-shaped text with total indifference to whether the world matches. When it’s right, it’s right as a side effect of its training data, not out of conviction. When it’s wrong, it’s wrong with exactly the same confidence and exactly the same beautiful prose.
That’s not a bug in the model. That is the model.
This is physics, and physics doesn’t care how you feel about it. Nobody wrapping their car around a tree is a fan of F=ma. It happens anyway. Learn the physics and work around it.
The model is not the system
This is where a lot of the public conversation goes sideways. People conflate models with systems.
The early chatbots were a model with a REPL loop bolted on top. Ask what the date was and you got some date well in the past, or the thing announced it was trained on data up to a point that had already receded over the horizon. All it could do was reach into the weights. Pure engine of bullshit.
Then we got serious. Every modern chatbot you deal with has grounding mechanisms. Ask it for arithmetic and it recognizes the shape of the request and calls a calculator instead of rolling dice on the statistics of digits. Ask it about a current fact and it runs a web search and feeds the results back into context, which is retrieval-augmented generation. The answer is now conditioned on both what’s in the weights and on real retrieved evidence.
That is not magic. That is engineering. It’s what it looks like when somebody stops hoping and starts building.
Put on your big-boy engineering pants
So if LLM models are bullshit engines, and we are building systems on top of them, what then must we do?
Start by recognizing that most of our modern technology stack is built on crap.
IP (Internet Protocol) is best-effort. It will drop your packet, duplicate it, deliver it out of order, and tell you about none of it. So we layered TCP (Transmission Control Protocol) on top: sequence numbers, acknowledgments, retransmission, backoff. Reliable stream, unreliable substrate.
An individual disk drive is a spinning platter of rust that will betray you faster than a politician’s campaign promise. So we apply engineering, RAID (Redundant Array of Inexpensive Disks), and now we have reliable storage built on unreliable disks. But that reliable storage lives in a building, and buildings burn down and lose power and get their fiber cut by a backhoe. So we apply more engineering: geo-replication.
DRAM flips bits when a cosmic ray wanders through. So: Error Correcting Code (ECC) memory.
Every layer you trust is standing on a layer that does not deserve trust. None of this is new. It’s why we pay engineers a lot of money. They are alchemists, turning leaden, unreliable components into services with gold-plated SLAs. (Service Level Agreements).
Here is the alchemical formula for building a reliable system out of unreliable components.
- Recognize the problem. The component is unreliable. Say it out loud, in a design review, in front of witnesses.
- Characterize the flaws. How exactly does it fail? What are the failure modes, how often, under what conditions, and what does failure look like from the outside?
- Detect the flaws. Build a mechanism that can tell, at runtime, that this specific failure just happened.
- Address the flaws. Have a strategy for when detection fires: correct, retry, degrade, or escalate to a human.
Every one of those four steps is hard in the LLM domain. Characterizing the failure modes of a stochastic natural-language component is far nastier than characterizing a bad disk sector. But they are all engineering problems, and engineering problems have engineering answers. Nobody is coming by with a pixie dust dispenser.
I stepped on this rake
I ran into all of this while building the AI Rosetta Stone project, part of my fellowship at Harvard’s Berkman Klein Center.
The system runs debates. Different AI points of view argue a topic against each other. I designed them to argue like lawyers, and arguing like a lawyer means you don’t just say things. You respond to the opposing party through a set of well-defined dialectic moves. Fifteen of them, grounded in computational dialectics and argumentation theory.
I started running debates. The engine hallucinated a sixteenth move.
My first reaction was the wrong one. I assumed the model had found a real gap in my taxonomy. Maybe my fifteen weren’t good enough. So I added the new move to the list. Then it hallucinated another one. I added that. Then another. I added that too.
Each addition made me more uncomfortable, because the fifteen weren’t arbitrary. They came from theory. I was letting a semi-random word generator with an excellent vocabulary and a vastly overdeveloped sense of confidence edit my epistemology, one shrug at a time. And it didn’t even work. It just kept inventing new moves.
So I did what everybody does. I went back to the prompt. I told the model, in plain language, that these were the only fifteen dialectic moves it could use. It hallucinated a new one. So I added a self-check at the end: review your dialectic move, and if it is not one of these fifteen, do not proceed. It reviewed its move, concluded it was smarter than me, and hallucinated anyway.
Prompt engineering has its place and it can make things better. For this problem it was the wrong tool.
Now look back at the last five paragraphs. Hallucinated. Hallucinated. Hallucinated. Hicks and his co-authors would tell me that word smuggles in a faculty that was never there, and they’d be right. Nothing misfired. The sampler did exactly what samplers do. The word makes the failure sound like an accident, which is precisely why it’s so comfortable to reach for, and precisely why I kept fiddling with prompts instead of building a boundary.
LLM output is user input
Then it clicked.
We have to treat LLM output the way we have always treated user input.
Only the worst engineers take user input and use it directly. Good engineers know that people make mistakes constantly. They fat-finger, they paste garbage, they lean on the keyboard. Beyond honest mistakes, you have actual bad actors crafting input to run a SQL injection against you. So we learned, decades ago and at considerable cost: never trust user input. Validate it at the boundary, before it touches anything that matters.
An LLM sits on the far side of exactly that kind of boundary. It is an untrusted producer of strings. The fact that the strings are eloquent is not evidence of anything at all. Trusting model output because it reads well is the same category error as trusting a form field because it’s spelled correctly.
We absolutely have to do this for LLM models but the reality is that general purpose LLM systems (Gemini, Claude, DeepSeek, etc) also hallucinate even when they try to be truth tellers.
What I actually built
I changed the system so that every LLM output is a JSON document conforming to a well-defined schema. Then I validate it in two passes.
First the form: does it parse, does it match the schema. Then the content, through a series of what I call mini-fixers. Small, specific validators that each know exactly one thing about the domain. For this field, the mini-fixer asks one question: is this value one of the fifteen moves?
When it isn’t, we go into a repair strategy. Compute the cosine similarity between the invented move and each of the fifteen legitimate ones. If the best match clears a confidence threshold, remap it. Usually the model meant one of mine and just reached for its own vocabulary. If it doesn’t clear the threshold, retry.
Detect, then correct. It’s the RAID playbook with a weirder failure mode.
The irony here is not lost on me. I spent a big chunk of my career arguing that a shell should pass structured objects instead of doing fragile, prayer-based text parsing. Twenty-six years later I’m having the same fight with a different unreliable component. Similar problem. Similar answer. Constrain the interface, validate at the boundary, and stop praying that your masking tape holds the plane together.
There is no pixie dust
LLMs look like magic. You have a big hard problem, you sprinkle some pixie dust on it, and the problem goes away.
There is no pixie dust. My system only looked like it was working, right up until I did real QA on it. That gap, between it demos beautifully and it survives inspection, is where an enormous number of AI projects are quietly living right now.
LLMs are still magic. I mean that. What these things can do is astonishing and I’m not walking any of it back. But if you’re building a system where it actually matters whether it works, you have to throw away the rose-colored glasses and do the unglamorous work of engineering.
The sooner executives understand that and stop laying off the engineers who do it, the better off everyone will be.
Because that’s the whole argument in one line. The model is an engine of bullshit. Your system does not have to be.
That’s a choice, and it’s yours. Skip the engineering and you’re shipping a bullshit system. Do the engineering and you’re not.
Cheers!
Jeffrey
Notes
Harry G. Frankfurt, On Bullshit (Princeton University Press, 2005). Originally published as an essay in Raritan in 1986.
Michael Townsen Hicks, James Humphries, and Joe Slater, “ChatGPT is bullshit,” Ethics and Information Technology 26, 38 (2024). doi.org/10.1007/s10676-024-09775-5
The Hicks paper drew published pushback, which is what should happen to a good provocation. See David Gunkel and Simon Coghlan, “Cut the crap: a critical response to ‘ChatGPT is bullshit’,” Ethics and Information Technology 27, 23 (2025). doi.org/10.1007/s10676-025-09828-3