I've just been handed a CSV file containing customer survey responses from three different regions. On the surface, it looks fine, but as soon as I load it, I can tell it's going to be a headache. We've got dates in three different formats, "Yes/No" answers written as "yes", "YES", and "Y", and a lot of trailing whitespace that's going to break my joins later.

The "Messy Room" Approach

My first instinct—and what most people do when they're in a rush—is to just start hacking away at the dataframe. Let's look at how I'd start this if I weren't thinking about a pipeline:

# The raw data
df <- read.csv("survey_data.csv")

# Fix the dates
df$date <- as.Date(df$date) 

# Fix the casing
df$region <- toupper(df$region)

# Trim the whitespace
df$comments <- trimws(df$comments)

# Handle the NAs in ratings
df$rating[is.na(df$rating)] <- 0

This works for a one-off script, but it's a nightmare for a real project. Why? Because I'm mutating the object in place over and over. If I run the script twice by accident, or if I want to change the order of operations, I have to restart the whole R session to get back to the original df. Plus, if I get another file from the European team tomorrow, I have to copy-paste this whole block of code. It's fragile.

Stop the Variable Bloat

I want to move toward a flow where data enters at the top and comes out clean at the bottom. I'm going to use the native pipe operator |> (or %>% if you're still heavily tied to magrittr) to chain these operations. Instead of modifying df five times, I'll treat it as a sequence of transformations.

library(dplyr)
library(lubridate)

cleaned_data <- df |>
  mutate(
    date = ymd(date),
    region = toupper(region),
    comments = trimws(comments),
    rating = coalesce(rating, 0)
  )

This is much better. I can actually read the logic: "Take the data, then mutate these four columns." But we're still just applying this to one specific object. If I have ten files to process, I'm still just copy-pasting the pipe. That's not a pipeline; that's just a shorter script.

Turning the Logic into a Machine

To make this a true pipeline, I need to decouple the cleaning logic from the data itself. I'll wrap that pipe inside a function. This way, the function becomes the "cleaning machine," and I can feed any dataset into it.

clean_survey_data <- function(data) {
  data |>
    mutate(
      date = ymd(date),
      region = toupper(region),
      comments = trimws(comments),
      rating = coalesce(rating, 0)
    ) |>
    filter(!is.na(date)) # I realized I can't trust rows with missing dates
}

# Now I can use it on any file
us_clean <- clean_survey_data(read.csv("us_survey.csv"))
eu_clean <- clean_survey_data(read.csv("eu_survey.csv"))
ap_clean <- clean_survey_data(read.csv("ap_survey.csv"))

I noticed something while testing this: the ymd() function from lubridate is great, but if the date is totally garbage, it just returns NA. By adding that filter(!is.na(date)) at the end of the pipe, I'm ensuring that the output of my pipeline is always "analysis-ready."

Handling the "What Ifs"

Here is where things usually break in production. What happens if the "region" column is missing entirely from one of the files? The whole function crashes. I don't want to write a dozen if statements; that ruins the elegance of the pipe. Instead, I'll add a quick check at the start of the function to ensure the required columns exist.

clean_survey_data <- function(data) {
  required_cols <- c("date", "region", "comments", "rating")
  
  if (!all(required_cols %in% colnames(data))) {
    stop("The input data is missing one of the required columns: ", 
         paste(setdiff(required_cols, colnames(data)), collapse = ", "))
  }

  data |>
    mutate(
      date = ymd(date),
      region = toupper(region),
      comments = trimws(comments),
      rating = coalesce(rating, 0)
    ) |>
    filter(!is.na(date))
}

Now, if I feed it a malformed CSV, R tells me exactly which column is missing rather than throwing a cryptic "column not found" error in the middle of a mutate call. This is the difference between a script that "works on my machine" and a professional data pipeline.

📋 Practical Task

Build a Product Catalog Standardization Pipeline

You have been given a raw dataset of product listings from different vendors. The data is inconsistent: some prices have currency symbols, some product names are in all caps, and some SKU numbers have accidental leading spaces.

Your Task: Create a function called standardize_products that implements the following pipeline:

• Validation: Check that the columns sku, product_name, and price all exist. If any are missing, use stop() to provide a clear error message.
• SKU Cleaning: Remove all leading and trailing whitespace from the sku column.
• Name Normalization: Convert the product_name to title case (use tools::toTitleCase() or a similar method).
• Price Conversion: The price column is currently a character string (e.g., "$12.99"). Remove the "$" sign and convert the column to a numeric type.
• Filtering: Remove any rows where the price resulted in an NA.

Test your function by creating a small dummy dataframe with at least one "dirty" row and one "missing column" scenario to ensure your validation logic works.