Questions to ask an AI vendor before you buy

Why this matters

Suppliers increasingly offer AI systems to government offices: chatbots for citizen queries, tools that read documents, systems that flag cases for review. Demonstrations are usually impressive. What matters is how the system behaves on your data, with your users, over several years. The questions below help a department test a supplier's claims before committing public money.

They do not replace your procurement process. Follow your procurement rules at every stage. Involve your finance, legal and information technology colleagues early. See procurement.

Data: where it goes and how it is used

• Where will our data be stored and processed? In which country and by which companies?
• Will our data or citizens' data be used to train or improve your models or anyone else's?
• Who on your side can see our data? Under what conditions?
• How long is data kept? Can we have it deleted on request with confirmation?
• What personal data does the system actually need? Can it work with less? See data minimisation.

Where data is held and which laws apply to it is a question of data sovereignty. Get the answers in writing.

Security

• How is data protected in storage and in transit?
• Who has administrative access? Is that access logged?
• How will you tell us about a security incident and how quickly?
• Has the system been tested against misuse, such as attempts to make it reveal data or ignore its instructions? See prompt injection.
• Do you rely on other suppliers, such as a cloud provider or another company's AI model? Who are they?

Accuracy and testing

• How was accuracy measured, on what data and by whom?
• Can we test the system on our own sample data before we buy?
• What kinds of mistakes does it make? How often did they occur in your own testing?
• Does it ever produce confident but false answers? How do users spot them? See hallucination.
• Has it been checked for unfair results for particular groups, regions or languages? See bias.
• How will performance be monitored after launch? What happens if it gets worse over time? See model drift.

Ask for evidence, not adjectives. A claim such as "highly accurate" means little without the test behind it.

Urdu and regional languages

• How well does the system handle Urdu in Nastaliq script, Roman Urdu and mixed Urdu and English?
• Which regional languages does it support? How was that tested?
• Were native speakers involved in testing?
• Can we test it with real examples of the language our users write and speak?

Many AI systems perform much better in English than in Urdu. Test this yourself rather than relying on a demonstration.

Human oversight and accountability

• Which decisions does the system make? Which does it only support?
• How can an officer review, correct or override its output?
• Is every output and action recorded so it can be checked later? See audit trail.
• Can the system explain why it produced a particular result?
• How can a citizen challenge a result that affects them?

A system that affects citizens should keep a named officer responsible. See human oversight and accountability.

Exit and lock-in

• If we stop using the system, how do we get all our data back, in what format and at what cost?
• Will you delete our data at the end of the contract and confirm it?
• Can another supplier take over without starting from scratch?
• What happens if your company is sold, closes or stops supporting the product?

Becoming dependent on one supplier is called vendor lock-in. Plan your exit before you sign.

Costs

• What is the full cost over the life of the contract, including licences, usage charges, training, support and upgrades?
• Are charges based on usage? What happens to the bill if usage grows?
• Which costs are fixed and which can change?
• What staff time will we need to run, check and maintain the system?

Warning signs

• Reluctance to let you test on your own data
• Vague answers about where data is stored or whether it is used for training
• Claims that the system never makes mistakes or needs no human checking
• No clear way to get your data back
• Pressure to sign quickly

To set clear rules for staff once a tool is in use, see our guide on writing an AI use note for your office.