AI systems make or influence decisions about loans, jobs, medical care and what people see. Building them responsibly means asking not only does it work? but for whom, with what failures, and who is accountable?
Core principles
Where bias comes from
| Source | Example |
|---|---|
| Historical bias | Past hiring data reflects past discrimination; a model trained on it reproduces it |
| Representation / sampling bias | Face datasets with few darker-skinned faces → higher error rates for those groups |
| Measurement bias | Using arrests as a proxy for crime, or health spending as a proxy for health need |
| Proxy features | Removing “race” doesn't help if postcode or name still encodes it |
| Feedback loops | Predictive policing sends officers where arrests were recorded, generating more records there |
Measuring fairness
# (group, true label, model prediction) for a toy loan-approval model; 1 = approve / would repay
data = [("A", 1, 1), ("A", 1, 1), ("A", 1, 0), ("A", 0, 1), ("A", 0, 0), ("A", 0, 0),
("B", 1, 1), ("B", 1, 0), ("B", 1, 0), ("B", 0, 0), ("B", 0, 0), ("B", 0, 0)]
def rates(group):
rows = [(y, p) for g, y, p in data if g == group]
selection = sum(p for _, p in rows) / len(rows) # P(approve)
positives = [p for y, p in rows if y == 1]
tpr = sum(positives) / len(positives) # P(approve | would repay)
return selection, tpr
a, b = rates("A"), rates("B")
print([round(x, 2) for x in a], [round(x, 2) for x in b]) # [0.5, 0.67] [0.17, 0.33] ← [selection rate, true-positive rate]
print(round(b[0] / a[0], 2)) # 0.33 ← disparate impact ratio (the "80% rule" flags < 0.8)
print(round(a[1] - b[1], 2)) # 0.33 ← equal-opportunity gap in true-positive rate| Criterion | Requires equal across groups | Intuition |
|---|---|---|
| Demographic parity | Selection rate P(ŷ = 1) | Same share of each group approved |
| Equal opportunity | True-positive rate P(ŷ = 1 | y = 1) | Qualified people approved equally often |
| Equalised odds | TPR and false-positive rate | Equal error rates of both kinds |
| Predictive parity / calibration | P(y = 1 | ŷ = 1) or scores mean the same | An approval means the same thing for everyone |
Safety and alignment
| Risk | What it looks like | Mitigation |
|---|---|---|
| Specification gaming / reward hacking | The system optimises the literal objective, not the intent (see Reinforcement Learning) | Careful objectives, human feedback, red-teaming |
| Distribution shift | A model trained on one hospital's scans fails at another | Monitoring, robust evaluation, domain adaptation |
| Adversarial examples | Tiny pixel changes flip a classifier; prompt injection hijacks an LLM agent | Adversarial training, input validation, least-privilege tools |
| Hallucination / overconfidence | Fluent but false answers | Retrieval grounding, calibrated uncertainty, citations, human review |
| Misuse | Deepfakes, fraud, disinformation, cyber-attacks | Usage policies, watermarking, access controls |
| Alignment of capable systems | Goals that diverge from human values; hard to correct once deployed | Interpretability, scalable oversight, evaluations before release |
Governance
| Framework | Key idea |
|---|---|
| EU AI Act (in force 2024, phased in to 2026–27) | Risk tiers: unacceptable (banned, e.g. social scoring), high-risk (hiring, credit, medical — strict obligations), limited (transparency, e.g. chatbots must disclose), minimal; extra duties for general-purpose models |
| NIST AI Risk Management Framework (2023) | Voluntary US framework: Govern, Map, Measure, Manage |
| OECD / UNESCO AI principles | International baselines for trustworthy, human-centred AI |
| Model cards & datasheets | Documentation of intended use, training data, evaluation by subgroup and limitations |
The Machine Learning notes' Responsible AI & Ethics topic covers the practical checklist for ML projects in more depth.
How can a model be biased if it never sees the protected attribute?
What is the alignment problem?
Name two fairness metrics and explain why they can conflict.