How do I read a massive text file without crashing my program?

I've seen a lot of developers start by using File.read("data.txt"). That works fine when you're dealing with a few kilobytes, but the moment you point your tool at a 2GB production log file, your RAM is going to vanish and your OS will probably kill the process. You don't want to load the whole file into memory at once.

The trick is to stream the file line-by-line. In Ruby, File.foreach is your best friend here. It opens the file, yields each line to a block, and then closes the file automatically when it's done. It's memory-efficient because it only ever holds one line in memory at a time.

# Instead of File.read, do this:
File.foreach("production.log") do |line|
  # Process each line individually
  puts "Processing: #{line}" if line.include?("CRITICAL")
end

What's the cleanest way to extract specific data from a messy string?

When you're building a processing tool, you're usually looking for a needle in a haystack. You could use split or slice, but once the text format gets slightly inconsistent, those methods break. Regular Expressions (Regex) are the way to go, but I highly recommend using named captures. They make your code much more readable because you aren't guessing what match[1] or match[2] refers to.

Let's say we're parsing a log that looks like [2023-10-01] ERROR: Database timeout. Here is how I'd handle that:

log_line = "[2023-10-01] ERROR: Database timeout"
pattern = /\[(?<date>.*?)\] (?<level>.*?): (?<message>.*)/

if (match = log_line.match(pattern))
  puts "Date: #{match[:date]}"     # 2023-10-01
  puts "Level: #{match[:level]}"   # ERROR
  puts "Message: #{match[:message]}" # Database timeout
end

It takes a second to get the syntax right, but it saves you from the "magic number" headache later on. Just a heads up: Regex can become a rabbit hole. If your pattern is getting too long to read, break it up or comment it heavily.

Should I just write a long script or wrap this in a class?

If this is a one-off script you'll run once and delete, a script is fine. But if you're building a "tool," you want it to be testable and reusable. I always suggest wrapping the logic in a class. This allows you to maintain state—like a counter for how many errors you've found—without relying on global variables, which are generally a bad idea in Ruby.

Here is a skeletal structure of how I'd organize a text processor. Notice how I separate the reading of the file from the processing of the data. This makes it way easier to write a test for the logic without actually needing a physical file on disk.

class LogSummarizer
  attr_reader :stats

  def initialize
    @stats = Hash.new(0)
  end

  def process_file(path)
    File.foreach(path) do |line|
      process_line(line)
    end
  end

  def process_line(line)
    # We only care about the log level (INFO, WARN, ERROR)
    if line =~ / (INFO|WARN|ERROR): /
      level = $1
      @stats[level] += 1
    end
  end
end

summarizer = LogSummarizer.new
summarizer.process_file("server.log")
puts summarizer.stats # {"INFO" => 120, "WARN" => 15, "ERROR" => 3}

📋 Practical Task

Exercise: The Server Error Frequency Analyzer

Your task is to build a tool that analyzes a system log and identifies which specific error messages are occurring most frequently.

Requirements:

• Create a class called ErrorAnalyzer.
• Implement a method that reads a file line-by-line using File.foreach.
• Use a regular expression to extract only the message part of lines that start with "ERROR:" (ignore INFO or WARN lines).
• Store the count of each unique error message in a hash.
• Create a method called top_errors that returns the top 3 most frequent error messages and their counts.

Test Data: Create a dummy file named system.log with at least 10 lines, ensuring some errors are repeated (e.g., ERROR: Connection Timeout appearing three times) so you can verify your sorting logic works.