I want to show you something that happens all the time in production environments. You build a model, the validation metrics look fantastic, you're about to ship it, and then—if you're lucky—someone asks, "But how does this actually perform for different groups of people?"

Most of the time, we look at global accuracy. But global accuracy is a mask. It can hide systemic failures that only affect a minority of your dataset. Let's walk through a scenario I ran into a while back with a loan approval dataset. I'll simulate the data here so we can play with it together.

The "Perfect" Model That's Actually Broken

# Let's set up a messy dataset
set.seed(42)
n <- 1000
data <- data.frame(
  income = rnorm(n, 50000, 15000),
  credit_score = rnorm(n, 650, 50),
  gender = sample(c("M", "F"), n, replace = TRUE),
  approved = sample(c(0, 1), n, replace = TRUE)
)

# Now, I'll inject some systemic bias. 
# Let's make the model 'prefer' men even when credit scores are identical.
data$approved <- ifelse(data$gender == "M" & data$credit_score > 630, 1,
                 ifelse(data$gender == "F" & data$credit_score > 670, 1, 
                 sample(c(0, 1), n, replace = TRUE, prob = c(0.8, 0.2))))

# Simple logistic regression to predict approval
model <- glm(approved ~ income + credit_score, data = data, family = binomial)
probs <- predict(model, type = "response")
preds <- ifelse(probs > 0.5, 1, 0)

# Check global accuracy
mean(preds == data$approved)

I run this, and I get an accuracy of around 70-75%. For a noisy financial dataset, that feels acceptable. I'd be tempted to stop here. But here's the thing: the model doesn't know about gender—I didn't even include it as a feature. I thought that meant the model was "fair." In reality, I just made the bias invisible.

Slicing the Performance

Whenever I suspect a model is behaving unfairly, I stop looking at the aggregate and start slicing. I want to see the False Negative Rate (FNR)—the people who should have been approved but were denied—broken down by group.

# Let's create a confusion matrix for each gender
performance_by_group <- data.frame(
  gender = data$gender,
  actual = data$approved,
  predicted = preds
)

# Calculate False Negative Rate: (Actual=1 & Pred=0) / (Actual=1)
library(dplyr)
bias_check <- performance_by_group %>%
  group_by(gender) %>%
  summarize(
    fnr = sum(actual == 1 & predicted == 0) / sum(actual == 1),
    accuracy = mean(actual == predicted)
  )

print(bias_check)

Now look at the output. You'll likely see that the accuracy is similar for both groups, but the FNR is significantly higher for women. This is the "smoking gun." The model is systematically denying qualified female applicants at a higher rate than qualified male applicants, even though the income and credit_score are the only things it's looking at. Wait, how is that possible if gender isn't a variable?

Hunting for Proxies

This is where it gets interesting. In the real world, bias rarely comes from a single "protected" column. It comes from proxies. Maybe zip_code correlates with ethnicity, or industry correlates with gender. In my simulated data, I hard-coded the bias, but in a real dataset, the model picks up on patterns in the training data that reflect historical human prejudice.

To detect this, I like to run a "shadow model." I'll train a second model that includes the protected attribute. If the protected attribute is a strong predictor of the outcome, it means the original model is likely using other variables as proxies for that attribute.

# Shadow model including the protected attribute
shadow_model <- glm(approved ~ income + credit_score + gender, data = data, family = binomial)
summary(shadow_model)

If the coefficient for gender is statistically significant (look for that p-value < 0.05), you've just proven that the target variable itself is biased. The model isn't "creating" the bias; it's efficiently learning the bias already present in your historical data. If you deploy this, you aren't just predicting loan success—you're automating historical discrimination.

Adjusting the Lens

So, what do we do? We can't just delete the biased data, and we can't just ignore the variable. One approach is to move from "Accuracy" to "Equal Opportunity." Instead of trying to get the same accuracy for everyone, we adjust our decision thresholds for different groups so that the False Negative Rates are equalized.

It's a bit of a balancing act. You're essentially saying, "I'm willing to accept a slightly lower global accuracy to ensure that a qualified person has the same chance of approval regardless of their group." In my experience, that's not just the ethical choice—it's the only way to build a model that is actually robust in the long run.

📋 Practical Task

Auditing the Mortgage Approval Model for Demographic Parity

You have been handed a pre-trained model and a test dataset containing mortgage applications. The dataset includes loan_amount, years_employed, age, and ethnicity.

Your task:

• Load the provided dataset and generate predictions using a logistic regression model (use loan_amount and years_employed as features).
• Calculate the False Positive Rate (FPR) — the rate at which unqualified applicants are approved — for each ethnicity group.
• Identify which group is being "over-approved" (highest FPR) and which is being "under-approved" (lowest FPR).
• Run a "shadow model" including the ethnicity variable and report whether the variable is a significant predictor (p < 0.05) of loan approval.

Submit your R code and a brief note identifying the group with the highest disparity in False Positive Rates.