I've seen this happen to plenty of engineers moving into the insurance space: you build a fraud detection model, run your evaluation script, and see a sparkling 99.2% accuracy. You're ready to ship it to production, feeling like a rockstar, until the business team tells you that the model hasn't actually flagged a single fraudulent claim in the last month.

Here is a piece of code that looks correct on the surface, but is fundamentally broken for fraud detection.

# Attempting to predict fraud (1 = Fraud, 0 = Legitimate)
library(randomForest)

# Assume claims_data has 10,000 rows; only 100 are actually fraud
model <- randomForest(fraud_status ~ ., data = train_data)
predictions <- predict(model, test_data)

# Evaluating the model
mean(predictions == test_data$fraud_status)
# Output: [1] 0.99

The 99% Accuracy Trap

The problem here isn't the code—it's the math of imbalanced classes. In fraud detection, the "positive" case (fraud) is usually a tiny fraction of the total data. If 99% of your claims are legitimate, a model can achieve 99% accuracy by simply guessing "Legitimate" for every single row. It's the easiest path to a high score, but it's completely useless for the business because the cost of a False Negative (missing a million-dollar fraud) is far higher than a False Positive (manually reviewing a clean claim).

If you run a confusion matrix on the code above, you'll likely see that the "True Positives" column is zero. You haven't built a fraud detector; you've built a "majority class" detector.

Balancing the Scales with Downsampling

To fix this, we need to force the model to pay attention to the fraud cases. One of the most straightforward ways to do this in R is downsampling the majority class. We aren't throwing away data to be lazy; we're creating a training set where the model sees an equal number of fraud and non-fraud examples, forcing it to learn the actual characteristics of fraud rather than just the probability of the class.

library(caret)

# Create a balanced training set
# We take all fraud cases and an equal number of non-fraud cases
fraud_cases <- train_data[train_data$fraud_status == 1, ]
legit_cases <- train_data[train_data$fraud_status == 0, ]
balanced_legit <- legit_cases[sample(nrow(legit_cases), nrow(fraud_cases)), ]

balanced_train <- rbind(fraud_cases, balanced_legit)

# Retrain the model on the balanced set
model_balanced <- randomForest(fraud_status ~ ., data = balanced_train)
predictions_balanced <- predict(model_balanced, test_data)

# Now check the Confusion Matrix
confusionMatrix(predictions_balanced, test_data$fraud_status)

Focusing on Precision and Recall

Once you've balanced the data, you'll notice your overall "Accuracy" actually drops. Don't panic. This is expected. You're now trading a few more False Positives for a massive increase in "Sensitivity" (or Recall). In fraud modeling, I always tell my team to ignore Accuracy entirely. Instead, look at the F1 Score—the harmonic mean of precision and recall.

Precision tells you: "Of all the claims I flagged as fraud, how many actually were?" Recall tells you: "Of all the actual fraud out there, how many did I successfully catch?" Depending on the business appetite for manual review, you'll tune your model to lean one way or the other, but you'll never do it by chasing a 99% accuracy metric again.

📋 Practical Task

Building a Balanced Fraud Classifier for Auto Insurance

You have been provided with a dataset insurance_claims_df containing 5,000 records. Only 2% of these records are marked as fraud (is_fraud == 1). The dataset includes features such as claim_amount, policy_age_months, and incident_severity.

Your task is to implement a robust fraud detection pipeline in R that avoids the accuracy trap. Complete the following:

• Split the data into a training set (80%) and a testing set (20%).
• Implement a downsampling strategy on the training set so that the model is trained on an equal number of fraud and non-fraud cases.
• Train a Random Forest model using the randomForest package.
• Evaluate the model on the original, imbalanced test set.
• Print the Confusion Matrix and calculate the Recall (Sensitivity) for the fraud class.

Your final output should demonstrate that the model is successfully identifying fraud cases, even if the overall accuracy is lower than the baseline majority-class guess.