We've talked about cleaning data, but there is a big difference between fixing a typo and deciding what to do with an outlier. In a real production pipeline, you can't just "eye-ball" a histogram and delete a few bars. If you do that, your analysis isn't reproducible, and your colleagues will tear your code apart during a peer review.

I prefer a systematic approach. Today, we're going to build a small utility to handle outliers in a dataset of city bike rental durations. Most rides are short, but you'll always find that one person who forgot to dock their bike for three days. That's an outlier that will skew your mean and ruin your model.

Creating our messy rental data

First, let's get some data. I'm going to simulate a small set of ride durations in minutes. I'll include some typical rides and a few "nightmare" scenarios where the bike was essentially stolen or forgotten.

# Trip durations in minutes
ride_data <- data.frame(
  trip_id = 1:12,
  duration = c(12, 15, 14, 18, 11, 1000, 16, 13, 22, 14, 500, 17)
)

print(ride_data)

Finding the fence posts with IQR

I don't like picking arbitrary numbers (like "anything over 60 minutes is an outlier") because the nature of the data changes. Instead, I use the Interquartile Range (IQR). It's a robust way to find the "middle 50%" of the data and set fences around it.

Anything outside 1.5 times the IQR is generally considered a candidate for removal or capping. Here is how I calculate those boundaries:

Q1 <- quantile(ride_data$duration, 0.25)
Q3 <- quantile(ride_data$duration, 0.75)
IQR_val <- IQR(ride_data$duration)

lower_fence <- Q1 - 1.5 * IQR_val
upper_fence <- Q3 + 1.5 * IQR_val

cat("Lower Fence:", lower_fence, "\nUpper Fence:", upper_fence)

The "blind deletion" mistake

Now, here is where I usually make my first mistake when I'm rushing. My instinct is to just slice the dataframe to remove the outliers immediately. It feels clean. I'll write the code to keep only the rows that fall within the fences.

# The 'fast and loose' way (My mistake)
cleaned_data <- ride_data[ride_data$duration >= lower_fence & 
                           ride_data$duration <= upper_fence, ]

Wait. I just looked at the row count. I lost 16% of my data in one line of code. In a small sample, that's fine. In a dataset of a million rows, deleting 16% of your observations just because they're "extreme" is a dangerous move. You might be deleting the most interesting part of your signal—like the "super-users" of your bike system.

Implementing a systematic capping strategy

Instead of deleting, I'll use a technique called "Winsorization" (or simply capping). Instead of throwing the data away, we cap the outliers at the fence values. This keeps the observation in the dataset but prevents it from pulling the average into orbit.

I'll write a small logic check: if a value is above the upper fence, set it to the fence; if it's below the lower fence, set it to the lower fence. Otherwise, leave it alone.

# A more professional approach: Capping
ride_data$duration_capped <- ifelse(ride_data$duration > upper_fence, 
                                     upper_fence, 
                                     ifelse(ride_data$duration < lower_fence, 
                                            lower_fence, 
                                            ride_data$duration))

# Let's compare the means
cat("Original Mean:", mean(ride_data$duration), "\n")
cat("Capped Mean:", mean(ride_data$duration_capped), "\n")

The difference is massive. By capping rather than deleting, we've preserved the sample size while stabilizing the statistic. This is the "systematic" part—we didn't guess the number 1000; the data told us where the boundary was based on its own distribution.

📋 Practical Task

Capping Extreme Values in a Taxi Fare Dataset

You have been handed a dataset of taxi fares that contains some obvious errors (e.g., fares of $500 for a 2-mile trip). You need to apply the IQR method to stabilize this data without deleting the rows.

Your Task:

1. Create a dataframe called taxi_fares with the following values: c(12.5, 15.0, 11.0, 14.2, 13.8, 250.0, 16.1, 12.0, 14.5, 800.0, 13.1, 15.5).
2. Calculate the 1st Quartile (Q1), 3rd Quartile (Q3), and the IQR.
3. Determine the upper_fence using the 1.5 * IQR rule.
4. Create a new column called fare_cleaned. If the original fare is greater than the upper_fence, replace it with the upper_fence value. Otherwise, keep the original fare.
5. Print the final dataframe to verify that the $250 and $800 values have been capped.