I’ve spent a lot of my career dealing with "dirty" data. Usually, it starts with a colleague saying, "The data is clean," and then five minutes into my analysis, I find a temperature reading of 4,000 degrees Celsius or a patient's age listed as -12. You can write a few filter() statements to catch these, but when you have fifty columns of sensor data, writing fifty manual checks is a recipe for a headache.

The Manual Slog

Let's start where I usually do—the manual way. I've got a small dummy dataset of industrial sensor readings. I want to make sure the temperature is between -50 and 150 and the pressure is between 0 and 500.

# Our messy sensor data
sensors <- data.frame(
  sensor_id = 1:5,
  temp = c(22.5, 180.1, 21.0, -60.5, 23.2),
  pressure = c(101.3, 105.2, -10.1, 102.4, 600.1),
  humidity = c(45, 47, 42, 110, 44)
)

# The manual check
bad_temp <- sensors$temp < -50 | sensors$temp > 150
bad_press <- sensors$pressure < 0 | sensors$pressure > 500
print(bad_temp)
print(bad_press)

This works for two columns. But look at that code. I'm repeating the same logic: value < min | value > max. If I add ten more sensors, I'm just copy-pasting lines and hoping I don't make a typo. That's where we stop being engineers and start being data entry clerks. I hate that.

Defining a Rulebook

I want to separate the logic of validation from the definition of the rules. What if I store the rules in a list? That way, if the hardware specs change and the temperature ceiling moves to 200, I only change one number in one place.

# A list where names match the column names
validation_rules <- list(
  temp = list(min = -50, max = 150),
  pressure = list(min = 0, max = 500),
  humidity = list(min = 0, max = 100)
)

Now I have a "rulebook." But how do I actually apply it? I could write a for loop, but in R, looping over columns often feels clunky. Let's try to write a small helper function that takes a vector and a rule, and returns a logical vector of "failures."

check_range <- function(x, rule) {
  return(x < rule$min | x > rule$max)
}

# Let's test it on one column
check_range(sensors$temp, validation_rules$temp)

Scaling the Validation

Okay, the function works. Now I need to apply this function to every column mentioned in my rulebook. This is where purrr::map or lapply comes in. I'll try lapply since it's base R, but the logic is the same: iterate through the rules, find the corresponding column in the data, and run the check.

# I'll use lapply to iterate over the names of the rules
results <- lapply(names(validation_rules), function(col_name) {
  rule <- validation_rules[[col_name]]
  data_vec <- sensors[[col_name]]
  
  # We'll return a named vector of indices that failed
  which(check_range(data_vec, rule))
})

names(results) <- names(validation_rules)
print(results)

Wait, this is better, but the output is a bit disjointed. It tells me which rows failed, but it doesn't give me a clean report. If I'm handing this to a QA team, they don't want a list of indices; they want to see the actual bad data. Let's adjust the function to return a data frame of the violations.

Building a Validation Report

Let's refine this. Instead of just returning the index, I'll create a function that returns a "violation" data frame containing the row ID, the column name, and the offending value. This is how you actually build production-grade validation.

validate_data <- function(df, rules) {
  all_violations <- list()
  
  for (col_name in names(rules)) {
    rule <- rules[[col_name]]
    vals <- df[[col_name]]
    
    # Find where the rule is violated
    fails <- which(vals < rule$min | vals > rule$max)
    
    if (length(fails) > 0) {
      all_violations[[col_name]] <- data.frame(
        row = fails,
        column = col_name,
        value = vals[fails],
        reason = paste0("Outside range [", rule$min, ",", rule$max, "]")
      )
    }
  }
  
  # Stack all the little data frames into one big report
  do.call(rbind, all_violations)
}

report <- validate_data(sensors, validation_rules)
print(report)

Now we're talking. Instead of hunting through the original data, I have a concise table of every single rule breach. If the report is empty, the data is clean. If it's not, I have a checklist of exactly what needs fixing. This pattern—defining rules in a configuration object and applying them via a wrapper function—is the gold standard for automating data hygiene.

📋 Practical Task

Exercise: Automating Quality Control for a Clinical Trial Dataset

You have been handed a dataset containing blood pressure readings from a clinical trial. The data is noisy, and you need to automate the validation process to identify "impossible" readings before the analysis begins.

The Dataset:

trial_data <- data.frame(
  patient_id = 1:6,
  systolic = c(120, 115, 300, 122, 40, 118),
  diastolic = c(80, 82, 85, 150, 78, -10),
  heart_rate = c(72, 75, 70, 210, 68, 74)
)

The Rules:

• systolic: Must be between 70 and 200.
• diastolic: Must be between 40 and 120.
• heart_rate: Must be between 40 and 160.

Your Task:

1. Create a named list called clinical_rules that stores the min and max for each of the three columns.
2. Write a function called generate_qc_report that takes a data frame and a rules list as arguments.
3. The function should iterate through the rules and return a single data frame containing the row number, the column name, and the offending_value for every instance where a rule is violated.
4. Run your function on trial_data and print the resulting report.