When you first start dealing with files or network data in C#, it's incredibly tempting to just "grab the whole thing." You're thinking about the data as a single unit—a document, a JSON response, or a log file—and the most intuitive method is to load that entire unit into a variable and get on with your logic. I've seen this countless times in code reviews, and while it works perfectly fine during development with a few test records, it's a ticking time bomb for production.

The temptation of loading everything at once

Imagine you're building a tool to scan a server's log file for a specific error code. If the log file is 50KB, File.ReadAllText("logs.txt") is great. It's one line of code, it's readable, and the performance hit is negligible. You've got the whole string in memory, and you can run a .Contains() or a Regex over it. It feels efficient because it's fast to write.

// The "Quick and Dirty" way
string logData = File.ReadAllText("server_logs.txt"); 
if (logData.Contains("CRITICAL_FAILURE")) 
{
    Console.WriteLine("Found an error!");
}

The problem here is that you're tying your application's memory usage directly to the size of the external file. This is what I call "linear memory scaling," and it's dangerous. What happens when that log file grows to 2GB? Your application will attempt to allocate a massive contiguous block of memory on the heap. Even if you have enough RAM, you'll likely hit an OutOfMemoryException or cause the Garbage Collector to freeze your app while it struggles to manage that giant string. You're essentially gambling that your input will always stay small.

Streaming data for constant memory overhead

The professional way to handle this is through Streams. Instead of treating the file like a single block of text, we treat it like a conveyor belt. You only ever hold a tiny piece of the file in memory at any given moment. By using a StreamReader, you can process the file line by line. Whether the file is 10KB or 10TB, your application's memory footprint stays almost exactly the same.

// The "Production Ready" way
using (FileStream fs = File.OpenRead("server_logs.txt"))
using (StreamReader reader = new StreamReader(fs))
{
    string line;
    while ((line = reader.ReadLine()) != null)
    {
        if (line.Contains("CRITICAL_FAILURE"))
        {
            Console.WriteLine("Found an error!");
            // We can break early if we only need the first occurrence
            break; 
        }
    }
}

Notice the using statements. Streams are "unmanaged resources," meaning they hold onto a handle to a file on the hard drive. If you don't dispose of them properly, that file stays locked, and other processes (or even your own app) won't be able to access it until the GC eventually gets around to cleaning up. I always recommend the using block to ensure the handle is released the microsecond you're done with it.

Trading convenience for stability

Now, is the streaming approach more verbose? Yes. You have to manage a loop and handle the reader. But you're trading a tiny bit of developer convenience for absolute stability. When you stream, you aren't loading the file into RAM; you're telling the OS to give you a window into the file. You read a chunk, process it, discard it, and move the window forward.

This mindset applies to more than just files. If you're downloading a large ZIP file from an API, don't use HttpClient.GetStringAsync(). Use GetStreamAsync(). If you're writing a large CSV to disk, don't build a giant string in a StringBuilder and then save it; use a StreamWriter to push lines to the disk as you generate them. Once you stop thinking about "data as a variable" and start thinking about "data as a flow," your apps will suddenly become much more resilient.

📋 Practical Task

Exercise: The Heavy Log Filter

You have been handed a legacy piece of code that crashes whenever it processes a log file larger than 500MB. The current implementation uses File.ReadAllLines(), which loads the entire array into memory.

Your Task: Rewrite the LogAnalyzer class to use a FileStream and StreamReader. The goal is to find and print every line that starts with the prefix "[ERROR]" without loading the entire file into memory at once.

// STARTING CODE (The "Broken" Version)
public class LogAnalyzer 
{
    public void PrintErrors(string filePath) 
    {
        // This is the line causing the OutOfMemoryException
        string[] allLines = File.ReadAllLines(filePath); 
        
        foreach (var line in allLines) 
        {
            if (line.StartsWith("[ERROR]")) 
            {
                Console.WriteLine(line);
            }
        }
    }
}

Implement the fix ensuring that:

1. You use a StreamReader to process the file line-by-line.
2. You properly dispose of the stream using a using statement.
3. The memory usage remains constant regardless of the file size.