A few years ago, I was mentoring a developer who was tasked with a seemingly simple data migration: we needed to calculate a "loyalty score" for every user in our database based on their order history and save it to a new column. He wrote a script that looked like this: User.all.each { |user| user.update(loyalty_score: calculate_score(user)) }. On his local machine with fifty test users, it ran in a blink. In production, with 800,000 users, the server hit a memory ceiling and the process was killed by the OS (the dreaded OOM killer). He spent half his afternoon wondering why the script just "disappeared" without an error message.

The Memory Wall

The problem here is how Active Record handles .all.each. When you call .all, Rails executes a single SQL query to fetch every single matching record from the database and instantiates a Ruby object for every single row. If you have a few hundred records, that's fine. But if you have hundreds of thousands, you're trying to shove a massive amount of data into your RAM all at once. Your application's memory usage spikes vertically, and if it exceeds the available system memory, the process crashes.

You might think, "Why not just use a limit and offset?" While you could manually write a loop to paginate through the records, it's tedious and error-prone. That's why Rails gives us find_each.

Efficient Iteration with find_each

find_each is designed specifically for this scenario. Instead of loading the entire table, it fetches records in small batches (defaulting to 1,000) and yields them to the block one by one. Once a batch is processed, Rails discards those objects and fetches the next batch. This keeps your memory footprint flat, regardless of whether you're processing ten thousand or ten million records.

# The dangerous way
User.where(active: true).each do |user|
  # This loads ALL active users into memory first
  user.send_newsletter_email
end

# The professional way
User.where(active: true).find_each(batch_size: 500) do |user|
  # This loads users in chunks of 500
  user.send_newsletter_email
end

I usually recommend explicitly setting the batch_size if the records are particularly "heavy" (e.g., they have many columns or large text fields). If 1,000 records are still pushing your memory limits, dropping it to 100 or 500 can keep the server stable. One important catch: find_each requires the records to be ordered by their primary key (usually id). If you try to add a custom .order() clause to your query, Rails will ignore it because it needs to use the ID to track where the last batch ended.

When to use find_in_batches

You'll often see find_in_batches mentioned alongside find_each. The difference is subtle but important. While find_each yields individual records to the block, find_in_batches yields the entire array (the batch) to the block.

I prefer find_in_batches when I need to perform a bulk operation that is more efficient than updating records one by one. For example, if you're using a bulk-insert gem or a raw SQL UPDATE ... WHERE id IN (...) statement, processing the array is much faster than hitting the database 1,000 times per batch.

# Use this when you need the array for bulk operations
User.find_in_batches(batch_size: 1000) do |group|
  user_ids = group.map(&:id)
  # Perform one single query for the whole group
  Order.where(user_id: user_ids).update_all(processed: true)
end

---

📋 Practical Task

Exercise: Bulk-Updating Legacy Product Categories

You have a Product model with 50,000 records. Some products have a legacy_category string that needs to be migrated into a new category_id integer column. Updating these one-by-one in a standard .each loop is causing the staging server to swap to disk and slow down significantly.

Your Task: Write a Ruby script (or a Rails migration block) that uses find_each to iterate through all products where legacy_category is not null. For each product, you must:

1. Look up the corresponding Category record based on the legacy_category name.
2. Update the product's category_id.
3. Set the legacy_category to nil.

Requirement: Ensure you use a batch_size of 500 to keep memory usage low. Your solution should not load all 50,000 products into memory at once.