Should I put everything in one giant script or split it up?

I've seen too many beginners write a 2,000-line script that does everything from API calls to final plotting. Please, don't do that. It's a nightmare to debug and an absolute slog to maintain. When you're building a real pipeline—let's say we're analyzing monthly e-commerce sales trends—you want to separate your concerns.

I usually break my pipelines into three distinct files: 01_clean.R, 02_analyze.R, and 03_visualize.R. This way, if you realize you messed up a date conversion in the cleaning phase, you don't have to re-run your expensive statistical models just to fix a typo. You can use the source() function in a main "controller" script to run them in order, but keeping the logic separate is the key to your sanity.

# main.R
source("scripts/01_clean.R")
source("scripts/02_analyze.R")
source("scripts/03_visualize.R")

How do I pass data between these steps without re-running the whole thing?

This is where a lot of people trip up. They try to keep everything in the global environment, which is risky because you might accidentally modify a variable and not remember how you got there. The professional way to handle this is by saving intermediate "snapshots" of your data.

I highly recommend using saveRDS() and readRDS(). Unlike CSVs, RDS files preserve R-specific data types—meaning your factors stay factors and your POSIXct timestamps stay timestamps. In our sales pipeline, the cleaning script should end by saving a "tidy" version of the dataset, and the analysis script should start by loading that specific file.

# At the end of 01_clean.R
saveRDS(cleaned_sales_data, "data/processed/sales_tidy.rds")

# At the start of 02_analyze.R
sales_data <- readRDS("data/processed/sales_tidy.rds")

How do I make the final output actually useful for someone else?

A pipeline isn't finished just because the code runs; it's finished when the stakeholder gets the answer. If you're just staring at a plot in your RStudio Plots pane, you haven't built a pipeline—you've just done an experiment. You need to automate the output.

I usually set up a dedicated output/ folder. Use ggsave() for plots and write_csv() for summary tables. I also like to include a timestamp in the filename so I can track how the results change over time as new data flows in. Here is a quick way to handle that in the visualization step of our sales project:

# 03_visualize.R
library(ggplot2)

plot_sales <- ggplot(sales_summary, aes(x = month, y = revenue)) + 
  geom_line() + 
  labs(title = "Monthly Revenue Growth")

# Create a dynamic filename with the current date
file_name <- paste0("output/sales_report_", Sys.Date(), ".png")
ggsave(file_name, plot_sales, width = 8, height = 6)

It feels like a bit more work upfront, but when your boss asks for the report three months from now, you can just hit "Run" and know exactly where the file is.

📋 Practical Task

Pipeline Build: Urban Air Quality Trend Analysis

You have been provided with a raw dataset air_quality_raw.csv containing daily pollutant levels (PM2.5, NO2) and dates for a city over two years. Your goal is to build a modular end-to-end pipeline that processes this data and produces a final summary plot.

Create three separate R scripts to accomplish the following:

• Script 1 (Cleaning): Load the raw CSV, convert the date column to a proper Date object, remove rows with missing values (NAs), and save the result as air_quality_clean.rds.
• Script 2 (Analysis): Load the .rds file, calculate the monthly average of PM2.5 levels using dplyr, and save this summarized data frame as pm25_monthly_summary.rds.
• Script 3 (Visualization): Load the summary .rds file and create a line plot of PM2.5 levels over time. Use ggsave() to export the plot as air_quality_trend.png in an output folder.

Finally, create a run_pipeline.R script that uses source() to execute all three scripts in the correct sequence.