Risk, oversight and accountability for AI decisions

Why this is a senior responsibility

When an AI system contributes to a decision about a citizen, the department remains responsible for that decision. Saying "the system decided" is not an answer a court, an ombudsman or a citizen should accept. Setting up the safeguards that make AI use defensible is a leadership task, because only senior officers can assign responsibility, fund oversight and decide that a use is too risky to proceed.

Classify the risk first

Risk classification means sorting each AI use by how much harm it could cause, so that safeguards match the stakes. A simple three-level approach works for most departments:

• Lower risk. Internal support with no direct effect on individuals: drafting, summarising public documents, organising information for staff who then decide. Basic guidance and spot checks are usually enough.
• Medium risk. Uses that shape what staff see or prioritise. Uses that touch personal data also belong here, such as sorting complaints or flagging files for attention. These need documented checks, monitoring and clear limits.
• High risk. Uses that affect a person's rights, entitlements, money, liberty, safety or official record. Uses that apply to vulnerable groups also count. These need the strongest safeguards and a senior decision to proceed at all.

When in doubt, classify upwards. For medium and high risk uses, require an impact assessment before launch: a written analysis of who could be affected, how and what will be done about it. Check your department's instructions and any government guidance on how such assessments should be done.

Make human oversight real

Human oversight is only meaningful if the person overseeing can actually catch and correct errors. Oversight fails quietly when:

• staff have too many cases to review properly and approve whatever the system suggests
• the reviewer cannot see why the system made its suggestion
• overriding the system takes more effort or carries more blame than accepting it
• nobody tracks how often staff disagree with the system

This tendency to defer to the machine is automation bias. To counter it, set realistic caseloads for reviewers, give them the information and authority to override, record overrides without penalty and review the override rate regularly. An override rate near zero is a warning sign, not a success.

Draw clear accountability lines

Accountability must sit with named people, not with a system or a supplier. For each AI use, write down:

• the senior owner, who approved the use and answers for it
• the operational owner, who runs it day to day and monitors performance
• the decision maker, who signs each decision that affects a citizen
• who can suspend it and on what triggers

Keep an audit trail: a record of what the system produced, what the human decided and why. Without it you cannot investigate a complaint, defend a decision or learn from a failure.

Provide redress

Citizens affected by an AI-assisted decision must be able to challenge it. Redress means:

• telling people that AI was involved (module 6)
• giving a reason for the decision that a person can understand
• offering review by a human who has authority to change the outcome
• correcting the record and fixing the cause when the system was wrong

Redress should not be harder for AI-assisted decisions than for others. If anything it should be easier, because errors can repeat across many cases before anyone notices.

Know when to stop

Set suspension triggers in advance: a rise in complaints, a pattern of errors affecting one group, a security incident or model drift found in monitoring. Decide who has authority to pause the system and make sure they will not need permission from the supplier to do so.

Try it

Choose a public service you know, such as issuing a certificate or handling a complaint. Imagine an AI tool that suggests a decision for each case. Using only public information, classify the risk, name the four accountability roles you would assign and write two triggers that would make you suspend the tool.

Check yourself

1. What are the three risk levels and what marks a use as high risk?
2. Give two signs that human oversight is failing.
3. What four things should redress include?

Answers: (1) Lower, medium and high; high risk uses affect rights, entitlements, money, liberty, safety or official records; uses that apply to vulnerable groups also count. (2) Any two of: reviewers with too many cases, no visibility of reasons, overriding being harder than accepting, an override rate near zero. (3) Telling people AI was involved, an understandable reason, human review with authority to change the outcome and correction of the record and cause.