Clinical AI: what managers must ask first

Why managers have a role

Decisions about diagnosis and treatment belong to qualified clinicians. But managers decide what gets bought, piloted, installed and paid for. If an AI tool that influences patient care is introduced badly, patients can be harmed, staff can be put in an impossible position and public trust can suffer. You do not need to judge the clinical science yourself. You do need to make sure the right questions are asked and answered by the right people before anything goes ahead.

This module gives you those questions. Use them whenever a vendor, donor, partner or colleague proposes an AI tool that touches diagnosis, triage, treatment or patient priority.

1. Evidence: does it work, for patients like ours?

• What is the evidence? Ask for published studies, not brochures. Ask whether anyone independent of the vendor has evaluated it.
• Who was it tested on? A tool tested on one population may perform differently on another. Ask whether it has been evaluated on patients similar to yours in age, sex, disease patterns and language.
• In what setting? Results from well-equipped hospitals may not hold where equipment, image quality, internet access or staffing differ.
• Compared with what? A claim of high accuracy means little without knowing what it was compared against and how often it misses cases or raises false alarms.

Be cautious of single impressive figures. Ask your clinical leads to review the evidence and tell you plainly what it does and does not show.

2. Regulation: is it allowed?

In many countries, software that supports diagnosis or treatment is regulated as a medical device and needs approval before clinical use. Ask:

• Who regulates this kind of product here and has it been through the required process?
• Is it approved for the exact use being proposed or only for something narrower?
• What does your department's guidance say about introducing new clinical technology?

If nobody can answer clearly, stop and seek advice from the appropriate legal and regulatory channels. Do not accept "it is approved abroad" as the end of the question. Never describe a tool as approved unless you have seen the approval.

3. Responsibility: who decides and who answers?

• The clinician remains responsible. The tool may suggest, but a qualified person decides. Make this explicit in written procedures.
• Can clinicians disagree with it? Staff must be free and expected to override the tool when their judgement differs, without pressure to follow it.
• Guard against over-trust. Busy staff may start accepting suggestions without checking, a tendency known as automation bias. Plan training and supervision to prevent it.
• Who answers if something goes wrong? Clarify accountability between the institution, the clinicians and the supplier before signing anything.
• Can patients be told? Patients should be able to know when AI played a part in their care and how to raise concerns or seek redress. Transparency supports trust.

4. Bias and fairness: who could be left behind?

AI tools can work less well for groups that were under-represented in their training data: women, children, older people, people with darker skin, speakers of particular languages or people from rural areas. This bias can deepen existing inequalities in care.

• Ask whether performance has been checked separately for different groups.
• Consider who might not be able to use the tool at all, for example because of language, literacy or access to a phone.
• Make sure a person can always reach a human route to care.

An impact assessment before introduction is a sensible way to think these questions through in writing.

5. Monitoring: how will we know if it stops working?

A tool that performed well at the start can get worse as patients, equipment or practice change. This is model drift.

• Agree in advance what will be measured, how often and by whom.
• Set up a simple way for clinicians to report errors and near misses.
• Decide in advance what level of problems would lead you to pause or withdraw the tool.
• Keep records so that decisions can be reviewed later.

A short checklist for your file

Before any clinical AI tool goes ahead, you should be able to point to written answers on: the evidence and who reviewed it; regulatory status for this exact use; who is responsible for decisions; how fairness has been checked; the monitoring plan; data protection arrangements (module 3); and how patients are informed. If any answer is missing, the proposal is not ready.

Try it

Imagine a supplier offers your hospital a free trial of a symptom-checking chatbot for the outpatient department. Using the five headings above, write down the three questions you would put to the supplier first and the one person in your institution you would ask to review the answers.

Check yourself

1. Why is "it is approved in another country" not enough?
2. What is automation bias and how can managers guard against it?
3. Name two things a monitoring plan for a clinical AI tool should include.

Answers: (1) Approval elsewhere does not show it meets the requirements here or works for your patients and setting. (2) The tendency to trust a machine's output too much; guard against it with training, supervision, clear procedures and freedom for clinicians to override. (3) Any two of: what is measured, how often and by whom, a way to report errors and near misses, a threshold for pausing or withdrawing, records for later review.