When you first decide to build a tool like grep, the instinct is usually to get the data into a format you can easily manipulate. In Rust, that usually means reading the target file into a String and then calling .lines() on it. It feels clean, it's only a few lines of code, and for a small configuration file or a short source code file, it works perfectly.

The "Just Load It All" Approach

I've seen a lot of developers start with something like this. They use std::fs::read_to_string to pull the entire file into memory, then iterate over the resulting string to find matches. It looks like this:

use std::fs;

fn search(pattern: &str, filename: &str) {
    let content = fs::read_to_string(filename).expect("Could not read file");
    for line in content.lines() {
        if line.contains(pattern) {
            println!("{}", line);
        }
    }
}

On the surface, this is great. You have the whole file in a buffer, and Rust's lines() iterator is incredibly efficient. But we're thinking like script writers here, not systems engineers. We're assuming the environment is friendly and the input is small.

The Memory Wall

Here is where this approach breaks: the "Log File Problem." Imagine you're using your tool to search through a 10GB production server log on a machine with only 8GB of RAM. The moment read_to_string is called, your program will attempt to allocate a contiguous block of memory for that entire 10GB string. It won't "slow down"—it will simply panic and crash with an out-of-memory error. Or, worse, it'll trigger heavy swap usage and freeze the entire OS.

As a rule of thumb, if you're building a CLI tool that processes files, you should almost never load the entire file into memory unless you have a very specific reason to do so (like needing to perform multiple passes over the data or doing complex regex look-aheads that span the whole document).

Streaming with BufReader

The professional way to handle this is to stream the file. Instead of pulling the file into your program, you create a small buffer and pull the file through that buffer piece by piece. In Rust, we use std::io::BufReader for this. This allows us to process the file line-by-line without ever holding more than one line in memory at a time.

Let's look at how we pivot the implementation. We switch from fs::read_to_string to fs::File::open, and then wrap that file handle in a BufReader.

use std::fs::File;
use std::io::{self, BufRead, BufReader};

fn search(pattern: &str, filename: &str) -> io::Result<()> {
    let file = File::open(filename)?;
    let reader = BufReader::new(file);

    for line in reader.lines() {
        // reader.lines() returns a Result
        let line = line?; 
        if line.contains(pattern) {
            println!("{}", line);
        }
    }
    Ok(())
}

Notice a few critical changes here. First, the memory footprint of this program is now constant. Whether the file is 10 kilobytes or 10 terabytes, the RAM usage stays roughly the same because we only ever store one line at a time. Second, we've shifted to returning an io::Result. In the naive version, I used .expect(), but in a real tool, you want to handle "File Not Found" or "Permission Denied" errors gracefully rather than crashing the whole process.

There is a slight trade-off: you're doing more frequent, smaller I/O calls. However, BufReader mitigates this by reading large chunks from the disk into an internal buffer and then handing you the lines from that buffer. It's the best of both worlds—the memory safety of streaming with the performance of bulk reads.

📋 Practical Task

Exercise: Adding Line-Number Tracking to the Grep Clone

Modify the streaming search function provided in the lesson. Instead of just printing the matching line, your tool should print the line number (starting at 1) followed by a colon and the line content (e.g., 42: This is the matching line).

You will need to incorporate a counter into the loop. Ensure that the line number increments for every line read, regardless of whether that line contains the search pattern or not.