Measuring whether AI actually helps

Measure time saved, quality, error rates and feedback from staff and citizens, so claims about AI in your office rest on evidence and not hope.

Why measure

It is easy to say an AI tool "saves hours" or "improves quality". It is harder to show it. Senior officers, auditors and the public deserve claims that are backed by evidence. Measurement also protects you: if a tool is quietly causing errors, you want to know before a citizen does.

Good measurement does not need a research team. It needs a baseline, a few honest measures and the discipline to report what you find.

Start with a baseline

You cannot measure improvement without knowing where you started. Before introducing a tool, record for the chosen task:

how long it takes, on average, from start to approved output
how many items are returned for correction
the common types of error
what staff and citizens currently say about it

Collect this over a normal period, not a quiet week or a crisis.

Measure time honestly

Time saved is the measure most often exaggerated. Count the whole task, not just the moment the AI produces text:

Writing the prompt and preparing inputs
Waiting for and reading the output
Checking facts, figures and references
Correcting and reformatting
Any extra review a senior officer now does

A draft that appears in ten seconds but takes thirty minutes to fix has not saved much. Record real times on a sample of items, using a simple log.

Measure quality and errors

Speed means nothing if quality falls. Use a short checklist applied the same way to AI-assisted and non-AI work:

Accuracy: are the facts, dates and references correct?
Completeness: is anything important missing?
Clarity: would the intended reader understand it?
Tone and format: does it meet office standards?

Track errors by type. AI tends to make particular kinds of mistake, such as invented references or confident wrong figures. See hallucination. Note errors caught at review and, more importantly, any that got through.

Watch for automation bias: reviewers checking AI work less carefully over time because it usually looks right. Spot-check a sample independently.

Gather feedback

Numbers do not tell the whole story. Ask the people involved:

Staff: a few short questions. Did it help? What went wrong? Do you trust the output? Would you keep using it?
Citizens: where outputs reach the public, look at complaints, follow-up queries and any feedback channel you already have. A rise in "I did not understand this letter" is a signal.

Keep feedback anonymous where possible, so people feel free to report problems.

Avoid exaggerated claims

When you report results, be careful and specific:

Report the range, not only the best case. "Between 5 and 20 minutes per item, depending on complexity" is more useful than "saves 20 minutes".
Do not scale up small samples. Results from three officers over two weeks do not show what will happen across a division for a year.
Separate novelty from lasting effect. Enthusiasm in week one often fades.
Count the costs. Training time, licences, extra review and IT support are part of the picture.
Say what you did not measure. Gaps are fine if stated clearly.
Never present vendor figures as your own results.

A useful sentence for any report: "In this pilot, on this task, we observed..." It keeps the claim the right size.

What to check before reporting

Was there a baseline collected the same way?
Were the measures chosen before the results came in?
Are errors that got through included, not only those caught?
Have you included negative feedback?
Would the claim survive questions from an auditor?