Hallucination: When AI Invents with Conviction

Episode 3 — Hallucination: When AI Invents with Conviction

BEGINNING — The problem nobody sees

Have you ever talked to an AI and it gave you such a good, articulate, and precise answer — that you didn't even think to verify it? Well, there you go. That's the problem.

Language models — ChatGPT, Claude, Gemini — have a characteristic that the industry calls "hallucination." The model generates completely false information with the same fluency, the same confidence, and the same structure as true information. It doesn't stutter. It doesn't hesitate. It doesn't put an asterisk saying "I just made this up." It delivers the lie with the same conviction with which it delivers the truth.

And this isn't a bug. It's not a flaw that will be fixed in the next version. It's a direct consequence of how these models work.

In the previous episode, I explained that an LLM is a machine for predicting the next word. It receives a sequence of text and calculates: what is the most probable word to come next? Then the next. And the next. And so on. It doesn't "know" anything. It doesn't consult a database of verified facts. It recognizes statistical patterns in the billions of texts it read during training.

When the sequence it's completing "asks" for data it doesn't have — a number, a date, a citation — it does what any good storyteller does: it invents something plausible. Something that fits the pattern. And it delivers it with complete naturalness.

Ask ChatGPT about a scientific article. Sometimes it will give you the title, authors, journal, year of publication. All perfect. Except that the article doesn't exist. The authors never wrote it. The journal exists, but that volume doesn't. Each detail is individually plausible — and the whole is a complete fabrication.

MIDDLE — The mirror: the human mind does the same thing

Now, this is where it gets interesting.

There is a concept in Vedānta that describes exactly this mechanism. Śaṅkarācārya, at the beginning of his commentary on the Brahma Sūtra, defines what he calls *adhyāsa* — superimposition. The definition is precise: *atasmin tad-buddhiḥ* — it is the cognition of something in something where it is not. It is seeing one thing and perceiving another. It is the rope on the dark ground that you swear is a snake.

Notice the structure: there is a real substratum (the rope), there is a projection (the snake), and there is the conviction that the projection is real. You don't see the rope and "imagine" a snake as a mental exercise. You *see* the snake. Your heart races. You jump back. The experience is completely real — until someone turns on the light.

The LLM does exactly this. There is a substratum (the training data), there is a projection (the generated response), and there is total conviction (perfect fluency). The model doesn't "invent" and tell you it invented. It presents the fabrication as fact. Just as your mind doesn't tell you it's projecting — it delivers the projection to you as direct perception.

And here's the point that matters: conviction is the problem. It's not the error itself. Everyone makes mistakes. What's dangerous is the error that comes packaged with certainty. AI doesn't say "maybe." The mind doesn't say "I'm projecting." In both cases, you need something external to break the illusion — verify the fact, turn on the light.

Think about how many times you were absolutely certain of a memory — and then discovered you were wrong. Or how many times you "knew" someone's motivation — and were completely mistaken. Cognitive biases, false memories, emotional projections. Cognitive psychology documents dozens of these mechanisms. We hallucinate all the time. AI has only made visible a mechanism that the human mind has always possessed.

The difference is that with AI, the mechanism became obvious. Nobody gets offended when we say ChatGPT hallucinates. But tell someone that their perception of the world is a superimposition — that they are seeing a snake where there is a rope — and observe their reaction.

MIDDLE (cont.) — Viveka: the antidote

Vedānta doesn't stop at diagnosis. Diagnosis without treatment is just distress. If the mind projects naturally, if hallucination is structural — how do you deal with it?

The answer is *viveka* — discernment. The ability to distinguish between what is *satya* (that which is real, that which sustains itself under investigation) and what is *mithyā* (that which appears but does not sustain itself, that which depends on something else to exist).

In the context of AI, *viveka* is simple: verify before trusting. Don't accept a beautiful answer as truth just because it's well-written. Check the source. Test the data. This isn't distrust — it's maturity. Just as a mature person doesn't believe everything they think just because they thought it.

In the context of life — and this is where Vedānta goes deeper than any machine learning paper — *viveka* is the investigation into the nature of what you take as real. If the mind projects onto the world in the same way that an LLM projects onto the prompt, the question becomes: what is the substratum? What is the rope beneath all the snakes?

This is the central investigation of Vedānta. And it's not an abstract investigation — it's extremely practical. Because as long as you confuse rope with snake, you suffer. As long as you accept hallucination as fact, you make wrong decisions. Discernment is not an intellectual luxury. It is an operational necessity. Both for those who use AI and for those who have a mind.

END — The human being is the original LLM

I will summarize what we've seen so far in the series:

• Episode 1: The LLM is a pattern machine. It learns by completing text, predicting the next word. It doesn't understand — it recognizes patterns.
• Episode 2: The context window is the limit of attention. What doesn't fit in the window doesn't exist for the model. Just like the human mind: what isn't in your attention now, for all practical purposes, doesn't exist.
• Episode 3 (today): The model hallucinates — it projects false information with total conviction. Exactly as the mind projects *adhyāsa* onto reality. The antidote in both cases is *viveka* — discernment, verification, investigation.

Notice the pattern: each AI mechanism mirrors a mechanism of the mind. And it's no coincidence. AI was built by human minds, trained on human text, optimized to appear human. It is, literally, a reflection of how we process information.

But there's a deeper layer. The LLM was trained by the data it received. All its behavior — including hallucinations — is a result of this training. In Sanskrit, we would call this data *saṃskāra* — the accumulated impressions that condition future behavior.

And then comes the question that will be the next episode:

If AI is the result of the data that trained it... what are you the result of? Who chose your training data? Who decided which patterns you would recognize, which responses you would give, which projections you would make? And more importantly: is it possible to retrain?

*Saṃskāra* — the impressions that form us. Next episode.

---

*Series: AI and Vedānta — Episode 3 of ?* *Previous episode: Context Window* *Next: Saṃskāra — Who Trained Your Mind?*