AI Risks and Responsible Use — Lesson 3
Bias, Fairness, and High-Stakes Decisions
Learning Objectives
- 1Identify where bias can enter AI systems.
- 2Recognize high-stakes decisions that require bias auditing.
- 3Apply fairness checks to AI-assisted decision processes.
How bias enters AI
AI bias enters through training data, model design, and deployment context. If a hiring model is trained on a decade of hiring data that favored certain demographics, the model will reproduce those patterns. If a loan approval model is trained on historical lending data that reflected discriminatory practices, it will perpetuate those practices.
Bias is not always obvious. It can manifest as: certain groups receiving systematically different outcomes, the model performing better for some populations than others, proxy variables correlating with protected characteristics (zip code as a proxy for race, name as a proxy for gender), or training data that underrepresents certain groups.
The important insight for business leaders is that AI does not create bias — it amplifies existing bias at scale. A biased human reviewer might affect dozens of decisions. A biased AI model affects thousands or millions.
High-stakes decision categories
AI used in decisions that materially affect people requires specific safeguards. High-stakes categories include: hiring and recruitment (screening resumes, ranking candidates), lending and credit (approval decisions, interest rates), insurance (risk assessment, claims processing), healthcare (diagnosis, treatment recommendations), and housing (tenant screening, pricing).
For these categories, many jurisdictions have or are developing regulations. The EU AI Act classifies certain AI applications as high-risk and requires documented assessments, human oversight, and bias testing. US state and city regulations are emerging around AI in hiring and lending.
Even without regulatory requirements, using biased AI in high-stakes decisions creates reputational, legal, and ethical risk. Organizations should audit AI systems used in these categories for disparate impact — systematically different outcomes across protected groups.
Practical fairness checks
Start with awareness: which AI systems at your organization make or influence decisions that affect people? For each one, who evaluated it for bias? What data was it trained on? Does anyone monitor outcomes for disparate impact?
Testing for bias requires comparing outcomes across groups. If an AI-assisted hiring process advances 40% of male applicants and 15% of female applicants with similar qualifications, that disparity needs investigation. The fix might be in the training data, the model, the features used, or the deployment context.
Human oversight is not a complete solution. If the humans reviewing AI recommendations have the same biases the AI learned from historical data, they may not catch biased recommendations. Structured review criteria and diverse review teams improve the effectiveness of human oversight.
Case Study
The delivery radius that discriminated
Situation
A delivery service used an AI model to optimize delivery routes and estimated delivery times. The model offered same-day delivery to affluent neighborhoods and next-day delivery to lower-income neighborhoods with similar distances, because the training data reflected historical delivery patterns that prioritized higher-value orders.
Analysis
The model optimized for business metrics (revenue per delivery) without considering fairness across communities. The result was effectively discriminatory service based on neighborhood income level, which correlated with race. The company faced public backlash and regulatory scrutiny.
Takeaway
Optimizing for business metrics without fairness constraints can produce discriminatory outcomes. When AI decisions affect access to services, audit for disparate impact across communities.
Reflection Questions
- 1. Does your organization use AI for any decisions that affect people (hiring, pricing, service levels)? Has anyone tested those systems for bias?
- 2. If a journalist investigated how your AI systems treat different demographic groups, would you be confident in what they would find?
Key Takeaways
- ✓AI does not create bias — it amplifies existing bias in training data at scale.
- ✓High-stakes decisions (hiring, lending, healthcare) require documented bias auditing.
- ✓Test for disparate impact by comparing outcomes across protected groups.
- ✓Human oversight alone is insufficient if reviewers share the same biases as the training data.