If you're coming into this lesson thinking that "cleaning data" is just a quick session of running na.omit() or deleting a few duplicate rows, I need to stop you right there. That's the most dangerous misconception I see mid-level analysts fall into. They assume that if the data is present, it's correct.

The Myth: "If it's not an NA, it's clean"

Let me show you why that thinking will wreck your analysis. Imagine you're looking at a column of "Country" names. You run a quick unique() check and you see this:

[1] "USA"        "U.S.A."     "United States" " us "       "USA "

To R, these are five different countries. If you group your data by this column to calculate an average, you aren't getting the average for the United States; you're getting five different, fragmented averages. None of these values are NA, but they are all "dirty." I've seen entire quarterly reports derailed because someone forgot that a trailing space (like in "USA ") makes a string unique.

The Reality: Normalization and Type Coercion

Real-world cleaning is less about "deleting the bad stuff" and more about "normalizing the inconsistent stuff." You have to move from a state of technical validity (the code runs) to logical validity (the data makes sense). This usually involves a three-pronged attack: trimming, case-folding, and coerced conversion.

First, always hit your strings with stringr::str_trim(). It’s a lifesaver. It strips out those invisible leading and trailing spaces that sneak in during manual data entry. After that, I usually force everything to lowercase or uppercase using tolower(). It doesn't matter if the data says "New York" or "new york"—it's the same city. Consistency is more important than capitalization in a dataset.

Then comes the tricky part: the "pseudo-numeric" column. You'll often find a column that should be numeric, but because someone typed "1,200$" instead of "1200", R imports the whole thing as a character vector. If you try to wrap that in as.numeric(), you'll get that dreaded warning: NAs introduced by coercion.

Here is how I actually handle that in a pipeline. I don't just coerce; I strip the noise first:

library(dplyr)
library(stringr)

cleaned_df <- messy_df %>%
  mutate(
    # Remove currency symbols and commas before converting to numeric
    price = str_remove_all(price, "[$,]"), 
    price = as.numeric(price),
    # Normalize the category names
    category = str_trim(category) %>% tolower()
  )

I want you to approach the upcoming exercise not as a "cleanup" task, but as a "standardization" task. Your goal isn't to make the dataset smaller—it's to make it predictable. If you find yourself writing ten different if_else statements for ten different misspellings, stop. Look for a pattern, use a regular expression, or create a lookup table to map the messy names to the correct ones.

📋 Practical Task

The Global Weather Station Restoration Project

You have been handed a dataset from a network of old weather stations. The data is a disaster. Your task is to transform this "raw" dataframe into a "gold" dataframe that is ready for statistical analysis.

The Dataset: Assume you have a dataframe called weather_raw with the following issues:

• station_id: Some IDs have leading zeros that were dropped, others have trailing spaces.
• temp_c: The temperatures are strings. Some use a comma as a decimal separator (e.g., "23,5") and some have "C" appended to the end (e.g., "23.5C").
• city: The city names are inconsistent (e.g., "LONDON", "London ", "london").
• date: The dates are in various formats: "2023-01-01" and "01/01/2023".

Your Requirements:

1. Standardize Cities: Ensure all city names are trimmed of whitespace and converted to Title Case (e.g., "London").
2. Fix Temperatures: Convert the temp_c column to a true numeric type. You must handle both the comma-decimal issue and the "C" suffix.
3. Normalize IDs: Ensure station_id is a character string with no trailing whitespace.
4. Unify Dates: Convert the date column into a formal Date object.

Deliverable: Write a dplyr pipeline that takes weather_raw and produces weather_clean. Your code should be robust enough that if a new row is added with "25,2C" in the temperature column, your pipeline handles it without creating an NA.