Checking AI output: errors, bias and over-trust

The skill that matters most

Anyone can get a draft from an AI assistant. The officer who adds value is the one who can tell when it is wrong. This module brings together the checks from earlier modules into one routine.

Three kinds of problem

1. Invented content. Hallucinations: rules, references, numbers, quotes or events that do not exist. They are fluent and specific, which makes them believable.
2. Distortion. Real content changed: a condition dropped, a figure misread, a position attributed to the wrong party, uncertainty turned into certainty.
3. Bias. Bias in what the AI suggests: assumptions about gender, region, language, class or religion that creep into wording, examples or recommendations.

The human problem: over-trust

The AI's mistakes are only half the risk. The other half is us. People tend to trust a machine's output more than it deserves, especially under time pressure. This is automation bias. A neat, confident draft at 6 pm on a deadline is exactly when it strikes.

Signs you may be over-trusting:

• you are checking less carefully than you would check a junior colleague's work
• you are accepting a figure or reference because it "looks right"
• you cannot say where a particular claim in your document came from

A checking routine

Before anything AI-assisted leaves your desk:

1. Facts: every name, number, date, rule, reference and quote checked against an original source. If you cannot find the source, remove it.
2. Meaning: compare with the source document. Are conditions, exceptions and attributions intact?
3. Fairness: read it as the citizen or colleague it concerns would. Is anything assumed about them because of who they are?
4. Fit: does it match your department's position, format and tone?
5. Ownership: are you willing to sign it and defend every line? If not, it is not finished.

Make checking easier

• Ask for sources and page numbers in your prompt (module 3).
• Ask the AI to list the claims in its own draft that most need checking.
• Keep AI drafts and final versions separate, so you can see what you changed.
• For high-stakes documents, ask a colleague to check the facts without seeing the AI draft.

Higher stakes, more checking

Scale your checking to the consequences. A draft email to a colleague needs a quick read. A summary going to the minister, a decision affecting a citizen's entitlement or anything that will be published needs the full routine and sometimes a second person. The National AI Policy 2025 calls for human oversight of critical operations for exactly this reason.

Try it

Ask an AI assistant to write a short paragraph on the legal framework for a topic you know well in your department, including references. Apply the five-step routine. Count the problems you find in each category.

Check yourself

1. What are the three kinds of problem in AI output?
2. What is automation bias and when is it most likely?
3. What should you do with a claim you cannot find a source for?

Answers: (1) Invented content, distortion and bias. (2) The tendency to trust machine output more than it deserves; most likely under time pressure. (3) Remove it.