Imagine you’re tasked with auditing a massive physical archive of records—say, ten thousand thick ledger books filling an entire warehouse. You have two ways to approach this. You could try to carry every single book back to your desk at once, piling them up until the desk collapses under the weight. Or, you could walk to the shelf, pull out one book, record the data you need, put it back, and move to the next one.

In Ruby, CSV.read is the equivalent of trying to carry the whole warehouse to your desk. It loads the entire file into your RAM (your "desk"). If the file is 50MB, you're probably fine. If it's 5GB, your application will freeze, the OS will start swapping to disk, and eventually, your process will crash with a "NoMemoryError". CSV.foreach, on the other hand, is the act of processing one row at a time. It keeps your memory footprint tiny, regardless of whether the file has ten rows or ten million.

The Memory Trap: Why CSV.read will kill your server

I've seen junior devs use CSV.read because it's convenient; it gives you an array of arrays that you can map, filter, and sort immediately. It feels like working with a standard Ruby array. But in production, that's a ticking time bomb. When you call CSV.read("large_data.csv"), Ruby allocates memory for every single string in every single cell of that file simultaneously.

If you're processing a file of e-commerce transactions to calculate total revenue, you don't actually need the 1st row and the 1,000,000th row in memory at the same time. You just need the current row's price, then you add it to a running total and move on.

Streaming with CSV.foreach

The foreach method is an iterator. It opens the file, reads one line, yields that line to the block, and then clears that line from memory to make room for the next one. Here is how I would handle a massive sales export:

require 'csv'

total_revenue = 0

# This doesn't load the file into RAM; it streams it.
CSV.foreach("massive_sales_export.csv", headers: true) do |row|
  # 'row' is a CSV::Row object, which acts like a hash
  total_revenue += row["price"].to_f
end

puts "Total Revenue: #{total_revenue}"

Notice how the memory usage stays flat throughout the entire process. Whether that file is 1MB or 10GB, your script will use roughly the same amount of RAM.

Trimming the Fat with Headers and Converters

When you're dealing with huge files, you want to spend as little time as possible transforming data inside the loop. Calling .to_f or .to_i on every single iteration is tedious and slightly slower. Ruby's CSV library has a built-in way to handle this via converters.

If you tell Ruby to use the :numeric converter, it will automatically attempt to convert any value that looks like a number into an Integer or Float before it even hits your block. I usually pair this with headers: true so I can access data by column name instead of counting indices (which is a nightmare to maintain when the CSV layout changes).

require 'csv'

# Using converters: :numeric saves us from manual .to_f calls
CSV.foreach("massive_sales_export.csv", headers: true, converters: :numeric) do |row|
  # row["price"] is already a Float here
  puts "Processing transaction #{row["transaction_id"]}..." if row["price"] > 1000
end

One last pro tip: if you're dealing with a file that's truly gargantuan and has weird encoding issues (which CSVs always do), add encoding: 'UTF-8' or invalid: :replace to your options. It'll save you from the dreaded Encoding::InvalidByteSequenceError halfway through a three-hour processing job.

📋 Practical Task

The Million-Dollar Transaction Analyzer

You have been given a CSV file named transactions.csv containing 100,000 rows of sales data. Each row has the following columns: transaction_id, category, amount, and date.

Your goal is to write a Ruby script that calculates two things without loading the entire file into memory:

• The Grand Total of all transaction amounts.
• The Most Frequent Category (the category that appears most often).

Requirements:

• Use CSV.foreach to ensure memory efficiency.
• Use headers: true to access columns by name.
• Use converters: :numeric to handle the amount conversion automatically.
• Use a Hash to track the counts of each category as you iterate through the file.