Data foundations: why data decides whether AI works

Why data comes first

Most AI projects that fail do not fail because the technology is weak. They fail because the data behind them is incomplete, inconsistent, out of date or unavailable. An AI system learns from data, works on data and is judged against data. If your records are poor, the system will be poor. It may also hide that fact behind confident outputs. Before approving any AI investment, a senior officer should ask a simple question: is our data good enough to support this?

Quality: is the data fit for the job?

Data quality is not an abstract idea. Ask your teams practical questions:

• Is it complete? Are there large gaps, such as missing fields, missing years or districts that never report?
• Is it accurate? How often are records wrong and does anyone check?
• Is it consistent? Are the same things recorded the same way across offices? Or do names, codes and categories vary?
• Is it current? How old is it and how quickly is it updated?
• Is it usable? Is it in a digital, structured form? Or is it locked in paper files, scanned images and free-text notes?

A dataset that is good enough for an annual report may be far from good enough for a system that makes suggestions about individual cases.

Hidden bias in records

Data records past decisions and past patterns of service. If some groups were served less, recorded less accurately or checked more often, the data carries that history. A system trained on it can repeat and amplify the pattern. This is one of the main sources of bias in AI.

Ask who is missing from the data, who is over-represented and whether past decisions recorded in it were themselves fair. If nobody can answer, you are not ready to use that data for decisions about people.

Ownership: who is responsible for the data?

In many organisations nobody clearly owns a dataset. It sits in a system, several teams use it and nobody is responsible for keeping it accurate. AI projects expose this quickly. For each dataset an AI project relies on, make sure there is:

• a named owner responsible for its quality and use
• a clear record of where it came from and how it is updated
• agreed rules on who may use it and for what purpose
• a process for correcting errors when they are found

Where data comes from another department or body, get the terms of use agreed in writing before work begins.

Protection: keeping citizens' data safe

Government holds sensitive information about citizens. Using it with AI raises the stakes. Data protection is not only a legal matter; it is the basis of public trust. Make sure your teams:

• follow the law and your department's instructions on personal data
• apply data minimisation: use only the data the task genuinely needs
• remove or protect identifying details wherever possible
• control who can access data and keep records of access
• never place classified or personal data in public AI tools

Remember that removing names does not always make data anonymous. Combinations of details such as age, location and occupation can identify a person.

What a senior officer can do

You do not need to fix data yourself. You can:

• ask for an honest assessment of data readiness before any AI proposal is approved
• fund data improvement as a project in its own right, not as an afterthought
• assign clear ownership for important datasets
• make data quality part of regular performance reporting
• treat "our data is not ready" as a valid and useful finding, not a failure

Often the best first investment is not in AI at all, but in better records. That investment pays off whether or not AI is used later.

Try it

Choose a public dataset published by any government body, such as statistics on a public website. Assess it against the five quality questions above. Note one thing that would make it unsuitable for an AI system that makes decisions about individuals.

Check yourself

1. Name four of the five data quality questions.
2. How can historical data introduce bias into an AI system?
3. What four things should be in place for each dataset an AI project relies on?

Answers: (1) Any four of: complete, accurate, consistent, current, usable. (2) It records past decisions and service patterns; if some groups were treated or recorded unfairly the system can repeat that. (3) A named owner, a record of origin and updates, agreed rules on use and a process for correcting errors.