AI in health systems: where it is used and where it stops

Why this matters for health managers

You will hear a great deal about AI in health: from vendors, from the media, from colleagues and from patients. Some of it is real progress. Some of it is marketing. As a manager you will be asked to approve purchases, allow pilots, answer questions from staff and respond when something goes wrong. You do not need to be a technical expert to do this well, but you do need a clear picture of what AI can and cannot do in a health setting.

This module gives you that picture in general terms. It does not describe any particular tool. Nothing here means that any tool is approved or in use in Pakistan. Always check your own department's instructions.

Four broad areas of use

Around the world, AI in health tends to fall into four broad areas.

1. Administration. Drafting letters and reports, summarising meeting notes, helping with duty rosters, forecasting stock needs and answering routine staff questions. This is the lowest-risk area, because a person checks the output before anything happens. Module 2 covers it.
2. Imaging support. Some systems use computer vision to flag possible findings on scans, X-rays or images of the eye or skin for a clinician to review. In many countries such software is treated as a medical device and must go through regulatory approval before clinical use.
3. Triage and symptom chatbots. A chatbot asks a person about symptoms and suggests what kind of care they might need. These can widen access to information, but they can also give wrong or unsafe advice with total confidence.
4. Records and information. AI can help transcribe notes, search large record systems or produce predictive analytics, such as estimates of which services will be busiest. These depend completely on the quality of the underlying records.

The limits you need to know

AI does not understand the patient. A system finds patterns in data it was trained on. It does not know the patient in front of a clinician, their history, their circumstances or what they have not said.

Performance does not travel automatically. A tool that performed well in one country, hospital or population may perform worse elsewhere. Differences in equipment, disease patterns, languages and record-keeping all matter. This is a form of bias and it can harm the very groups who are least represented in the training data.

General chatbots are not medical tools. A general AI assistant can produce fluent, specific medical-sounding text that is simply wrong. These hallucinations are especially dangerous in health, where a confident wrong answer can delay real care.

Performance can drift. A system that worked at the start can get worse as conditions change, a problem known as model drift. Any use of AI near patient care needs ongoing monitoring, not a one-off check.

Responsibility does not move to the machine. Whatever a tool suggests, the clinician remains responsible for clinical decisions and the institution remains responsible for how the tool is used. Human oversight and clear accountability are not optional extras.

A simple way to sort what you hear

When someone proposes an AI use, ask one question first: what happens if it is wrong?

• If a wrong output means a badly worded draft that a person corrects, the risk is low and good office habits are enough.
• If a wrong output could affect a patient's diagnosis, treatment, priority or access to care, the risk is high. That needs clinical leadership, evidence, regulatory checks and monitoring before anything goes ahead. Module 4 covers the questions to ask.
• If the use involves patient records in any form, data protection comes first. Module 3 covers this.

Be wary of claims

Vendors and enthusiasts may describe tools with phrases like "as accurate as a doctor" or "proven in hospitals". Treat these as claims to test, not facts. Ask where the evidence was published, which patients it was tested on and whether anyone independent has checked it. If the answers are vague, so is the evidence.

Try it

Find a public news article about an AI tool used in health anywhere in the world. Using the four areas above, decide which area it falls into. Then write down the answer to "what happens if it is wrong?" and list two questions you would want answered before allowing something similar in your setting.

Check yourself

1. Name the four broad areas where AI is used in health systems.
2. Why might a tool that worked well in another country perform worse in your setting?
3. What single question helps you sort low-risk from high-risk uses?

Answers: (1) Administration, imaging support, triage and symptom chatbots, records and information. (2) Differences in population, equipment, disease patterns, language and record-keeping mean performance does not travel automatically. (3) What happens if it is wrong?