Bias, fairness and who bears the error

Every system has an error rate. The question that matters is who the errors land on, and whether anyone measured before deployment.

intermediate10 minChecked 2026-09-06

What you will be able to do

Ask the right questions about a system's error distribution, and recognise when an aggregate accuracy figure is hiding something.

Before you start

  • None.

Aggregate accuracy conceals

A system reported as 95 per cent accurate may be near-perfect for one group and substantially worse for another. The headline number is an average across a test set, and the composition of that set determines what it hides.

The influential demonstration of this evaluated commercial gender-classification systems and found error rates that differed sharply across skin tone and gender — highest for darker-skinned women, lowest for lighter-skinned men. The finding was not that the systems were useless. It was that a single accuracy figure had concealed who they failed.

Where bias enters

Training data that under-represents a group. Historical data encoding past discrimination as signal. Labels applied inconsistently. A target variable that stands in poorly for what you actually care about. Deployment in a population unlike the one it was tested on.

None of these require anyone to intend harm, which is precisely why measurement matters more than intent.

What to ask before trusting a system

What was it evaluated on, and did that population resemble mine? Is performance reported by subgroup? What happens when it is wrong, and to whom? Is there a route to challenge a decision? Who is accountable?

Model cards and internal auditing frameworks exist to make these questions answerable. Their absence is itself an answer.

What good looks like

  • You ask for disaggregated performance rather than accepting an average.
  • You know what happens when the system is wrong and who absorbs it.
  • There is a documented route to challenge an outcome.

What it cannot do

  • Fairness has multiple mathematical definitions that cannot all be satisfied at once; the choice between them is a values decision, not a technical one.
  • Findings about specific commercial systems are dated and those systems change.

When it goes wrong

A vendor cites one accuracy figure.
Ask for performance by subgroup and for the composition of the evaluation set. A refusal is informative.
Assuming a general-purpose model is fair by default.
It reflects its training data. Test on your own population before relying on it.

Privacy and data handling

Fairness testing often requires demographic data, which is itself sensitive. It needs a lawful basis and careful handling, not an informal spreadsheet.

Sources4 referenced
  1. Gender Shades: intersectional accuracy disparities in commercial gender classification

    Buolamwini & Gebru, Proceedings of Machine Learning Research (2018) · peer-reviewed study · link checked 2026-09-06

  2. Model cards for model reporting

    Mitchell, Wu, Zaldivar, Barnes, Vasserman, Hutchinson, Spitzer, Raji & Gebru, FAccT (2019) · peer-reviewed paper · link checked 2026-09-06

  3. Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing

    Raji, Smart, White, Mitchell, Gebru, Hutchinson, Smith-Loud, Theron & Barnes, FAccT (2020) · peer-reviewed paper · link checked 2026-09-06

  4. AI Risk Management Framework

    NIST · standards framework · link checked 2026-09-06

Links are checked at build time. If one has moved, tell us.

Written by
BLACK LDN
Last checked
2026-09-06
Content state
editorial

Product behaviour changes. If something here no longer matches the documentation, send a correction.

Read next

How these tools work8 min

Why AI can sound certain and be wrong

Confabulation is not a bug that will be patched out next quarter. It follows from the design, and knowing that changes how you work.

beginner

Checked 2026-09-06

Using it responsibly9 min

Copyright, provenance, work and opportunity

The legal position is genuinely unsettled in places. The practical position is clearer than the discourse suggests.

intermediate

Checked 2026-09-06

Using it responsibly9 min

Privacy and sensitive information

The question is not whether these tools are safe. It is what you are permitted to put into them, and what happens to it afterwards.

beginner

Checked 2026-09-06

Next: Copyright, provenance, work and opportunity