How to run a small, safe pilot of an AI tool

Test an AI tool on one low-risk task in your office, measure it against current practice and make a clear decision on whether to continue.

Why pilot first

An AI tool that looks impressive in a demonstration may behave very differently on your office's real work. A small pilot lets you find out cheaply and safely. It gives you evidence for a decision, instead of enthusiasm or fear. It also shows staff that new tools will be tested properly before anyone depends on them.

A pilot is not a launch. It is a limited, time-bound test with a clear question: does this tool do this task better than how we do it now?

Step 1: choose a low-risk task

Pick one task that is frequent enough to measure but where mistakes are easy to catch and cheap to fix. Good candidates:

first drafts of routine replies that an officer always reviews
summarising long internal reports for a briefing
turning meeting notes into draft minutes
producing a plain-language version of a standard notice

Avoid tasks that directly decide a citizen's rights, payments, licences or complaints. Avoid anything involving classified or sensitive personal data unless the tool is approved for it. See data protection.

Step 2: write down how it works today

Before you start, record current practice for the chosen task. How long does it usually take? Who does it? How often is it sent back for correction? What do recipients complain about? Without this baseline, you cannot tell whether the tool helped.

Step 3: set your measures

Decide in advance what success looks like. Keep it to a few measures you can actually collect:

Time: minutes per item, including checking and correcting
Quality: errors found at review, using a simple checklist
Rework: how many items needed significant changes
Staff view: a short feedback form at the end
Risk events: any case of sensitive data entered or a wrong output nearly sent

Set a threshold for each. For example: "continue only if time falls and errors do not rise." See Measuring whether AI actually helps.

Step 4: get approval

Put a short note to the competent authority. Cover the task, the tool, the data it will see, the staff involved, the duration, the measures and how you will stop if something goes wrong. Involve your IT or information security focal person early. If the tool costs money, follow your procurement rules. See procurement.

Record the approval on file. A pilot without approval is just shadow AI with a plan.

Step 5: run it, with a human on every output

Keep the pilot small: a few staff, a few weeks.
Brief participants on what to enter and what never to enter.
Every AI output is checked by an officer before use. See human in the loop.
Ideally, run the old method alongside for some items so you can compare like with like.
Log every item: time taken, errors found, changes made.
Note problems as they happen, not from memory at the end.

Step 6: compare and decide

At the end, put the pilot results next to your baseline. Be honest about what the numbers show and what they do not. A small pilot gives you an indication, not proof.

There are three sensible outcomes:

Stop. The tool did not help enough or the risks were too high. This is a successful pilot: you learned something without harm.
Adjust and test again. The tool showed promise but needs better instructions, training or a different task.
Expand carefully. Results were clearly positive. Seek approval to widen use, with written guidance and continued monitoring.

Write a short pilot report for the file, including what went wrong. Share it with colleagues who may be considering the same tool.

Warning signs

The vendor runs the pilot and reports the results
Measures are chosen after the results are in
Staff feel unable to report problems
The pilot quietly becomes permanent without a decision