You're likely at the point in your project where you've cleaned your strings and handled your dates, but you've hit the "NA Wall." You run a linear model or a correlation matrix, and R just returns NA for everything, or worse, it silently drops 30% of your rows because of na.omit(). Your first instinct is usually to just fill those holes with the average of the column so the code stops crashing. Let's look at why that usually breaks your analysis.

# A typical "quick fix" that causes problems
set.seed(42)
data <- data.frame(
  experience = c(1, 2, 3, 15, 18, 20, NA, NA, 22, 1),
  salary = c(45000, 48000, 52000, 120000, 130000, 140000, 125000, 135000, 150000, 44000)
)

# The "Rookie" Imputation: Global Mean
data$experience[is.na(data$experience)] <- mean(data$experience, na.rm = TRUE)

print(data)

The Bias of the Global Mean

If you run the code above, you'll notice something jarring. The missing values for experience were likely for the high-earners (since they align with salaries over 120k), but we just filled them with the global average of roughly 11 years. Now, your dataset contains people making 130k with only 11 years of experience, which completely warps the relationship between seniority and pay. You haven't "fixed" the data; you've introduced a systematic bias that will make your regression coefficients lie to you.

Switching to Median for Skewed Distributions

Before jumping to complex algorithms, check your distribution. If your data is skewed (like income or house prices), the mean is a terrible choice because outliers pull it away from the "typical" value. The median is far more robust. I generally lean toward the median for any financial or count data unless I have a specific reason not to.

# A slightly better approach for skewed data
median_val <- median(data$experience, na.rm = TRUE)
data$experience[is.na(data$experience)] <- median_val

This is safer than the mean, but it still treats every missing value the same, regardless of other variables. It's a "blind" fill.

Using MICE for Predictive Imputation

When the relationship between variables is strong, you shouldn't be guessing a single number. You should be predicting the missing value based on the other available data. This is where MICE (Multivariate Imputation by Chained Equations) comes in. Instead of filling a hole with a constant, MICE looks at the rest of the row and says, "Given that this person makes 130k, their experience is likely closer to 17 than to 11."

# You'll need the mice package for this
# install.packages("mice")
library(mice)

# MICE creates multiple imputed datasets to account for uncertainty
# we'll use 'pmm' (predictive mean matching) which is great for most numeric data
imputed_data <- mice(data, m=5, method='pmm', seed=500)

# Complete the data by picking one of the imputed sets
final_df <- complete(imputed_data)

print(final_df)

I prefer MICE because it preserves the variance of the dataset. If you fill 100 rows with the exact same mean value, you've artificially shrunk your standard deviation, which can lead to "significant" p-values that are actually just artifacts of your imputation. MICE keeps the data "natural."

📋 Practical Task

Repairing a Messy Sensor Dataset with MICE

You have been handed a dataset from a weather station where the temperature sensor failed intermittently. You cannot simply drop the rows because the timestamps are critical for the time-series analysis. You also cannot use the global mean because the temperature fluctuates wildly between the "Day" and "Night" cycles.

Your Goal: Use the mice package to impute the missing temp values based on the humidity and pressure columns.

# Setup the broken dataset
weather_data <- data.frame(
  temp = c(22.1, 21.5, NA, 15.2, 14.8, NA, 23.0, 22.8, NA, 16.1),
  humidity = c(45, 47, 46, 80, 82, 85, 44, 42, 48, 78),
  pressure = c(1012, 1011, 1013, 1005, 1004, 1006, 1012, 1011, 1014, 1007)
)

# 1. Load the mice library
# 2. Run the mice() function on weather_data using 'pmm'
# 3. Use complete() to generate the final data frame
# 4. Print the final data frame to verify no NAs remain