How chatbots work: why they sound sure even when wrong

A different question

Our guide What AI actually is explains what AI can and cannot do. This guide looks at one thing only: what happens inside a chatbot when you type a question and an answer appears.

Knowing this changes how you read the answers. It is the single most useful thing to understand about chatbots.

It writes one piece at a time

A chatbot is built on a large language model. During training, the model read a huge amount of text: books, websites, articles and conversations. From this it learned patterns. It learned which words tend to follow which other words, in which order and in which kind of writing.

When you ask a question, the model does not look up a fact in a book. It writes its answer one small piece at a time. At each step it asks itself, in effect: "Given everything so far, what is the most likely next piece of text?" Then it adds that piece and repeats, very quickly, until the answer is complete.

Think of a friend who has read thousands of letters and can finish any sentence you start. They are very good at sounding right. That is not the same as knowing they are right.

Why it sounds so confident

The model has learned how confident, helpful answers are written. So it writes in that style, whether or not the content is correct.

• It has no built-in sense of doubt. A true sentence and a false sentence can be equally likely as text. The model produces both in the same calm tone.
• It fills gaps. If it does not have the information, it often produces something that looks like the right kind of answer: a plausible name, date, figure or reference. This is called a hallucination.
• It tends to agree with you. Chatbots are tuned to be helpful and polite. If your question assumes something false, the chatbot may go along with it. This is called sycophancy.
• Its knowledge stops at a date. Each model learned from text up to a certain point, its knowledge cutoff. Unless it searches the web, it may not know about recent prices, rules or events.

Some chatbots can search the internet or read a file you give them. This helps, but they can still misread or mix up what they find.

What it is good at

Because it works with patterns of language, a chatbot is usually strong at tasks where the shape of the writing matters most:

• rewriting a message more politely or more simply
• explaining a common idea in plain words
• suggesting a structure for a letter or a plan
• giving you a list of questions to think about

It is weakest where one exact fact matters: a phone number, a fee, a date, a law, a medicine dose or a quote from a real person.

What this means for you

1. Treat answers as a first draft, not the final word. Use them to start thinking, not to finish.
2. Check any fact that matters against an official website, a document or a qualified person.
3. Ask for sources, then open them yourself. A chatbot can produce a source that does not exist. Only trust a link you have opened and read.
4. Do not be persuaded by tone. A smooth, confident answer is no more likely to be true than a hesitant one.
5. Ask the question neutrally. Instead of "Why is this medicine safe for children?", ask "Is this medicine safe for children? What should I check?"

Warning signs in an answer

• Very precise numbers with no source
• Names of people, offices or reports you cannot find anywhere else
• Different answers when you ask the same question twice
• Agreement with everything you say, even when you change your mind

None of these means the chatbot is useless. They mean a human needs to check.

Next, read Using AI safely: what never to type into a chatbot.