When you're building tools for real-world environments, you'll often find that the data isn't sitting in a tidy database. It's in a text file, a CSV, or a massive log dump. Today, we're going to build a processor that parses a server log file and generates a summary of how many errors occurred for each specific service. It's a classic pattern: Read → Parse → Transform → Write.

Defining our Log Structure

First, I don't want to pass around raw strings throughout the app. That's a recipe for bugs. I'll start by defining a simple record to represent a log entry. I'm using a record here because this data is immutable; once a log is written, it doesn't change.

public record LogEntry(DateTime Timestamp, string Level, string Service, string Message);

Now, I need a way to turn a line of text—something like "2023-10-01 12:00:01|ERROR|PaymentGateway|Timeout occurred"—into that record. I'll write a small helper method to handle the splitting logic.

The Naive Loading Mistake

Here is where I usually start, and where I often make a mistake in the early stages of a project. I'll write a method to read the file and parse the lines. At first glance, this looks clean:

public List<LogEntry> LoadLogs(string filePath)
{
    var lines = File.ReadAllLines(filePath); // The mistake is here
    var entries = new List<LogEntry>();
    
    foreach (var line in lines)
    {
        var parts = line.Split('|');
        entries.Add(new LogEntry(DateTime.Parse(parts[0]), parts[1], parts[2], parts[3]));
    }
    return entries;
}

I've done this a dozen times, and it works fine for a 10KB file. But imagine this is a production server log that's 4GB. File.ReadAllLines loads the entire file into a string array in RAM. Your app will throw an OutOfMemoryException before it even starts parsing. It's a classic "it worked on my machine" bug.

Streaming the Data for Efficiency

To fix this, I need to switch to a streaming approach. Instead of loading everything, I'll use File.ReadLines. Unlike ReadAllLines, this returns an IEnumerable<string>, meaning it reads one line at a time as the loop requests it. This keeps the memory footprint tiny regardless of file size.

public IEnumerable<LogEntry> StreamLogs(string filePath)
{
    foreach (var line in File.ReadLines(filePath))
    {
        var parts = line.Split('|');
        if (parts.Length == 4)
        {
            yield return new LogEntry(DateTime.Parse(parts[0]), parts[1], parts[2], parts[3]);
        }
    }
}

I added a quick length check too. Real logs are messy; you'll inevitably hit a blank line or a corrupted entry that would otherwise crash your Split logic.

Aggregating and Writing the Report

Now that we have a memory-efficient stream of data, we can use LINQ to do the heavy lifting. I want to filter for only "ERROR" levels and then group them by the service name.

public void ProcessErrorReport(string inputPath, string outputPath)
{
    var errorCounts = StreamLogs(inputPath)
        .Where(e => e.Level == "ERROR")
        .GroupBy(e => e.Service)
        .Select(g => new { Service = g.Key, Count = g.Count() })
        .OrderByDescending(x => x.Count);

    using var writer = new StreamWriter(outputPath);
    writer.WriteLine("Service Error Report");
    writer.WriteLine("-------------------");
    
    foreach (var item in errorCounts)
    {
        writer.WriteLine($"{item.Service}: {item.Count} errors");
    }
}

By combining IEnumerable and StreamWriter, the data flows from the disk, through our filter, and back to the disk without ever loading the whole dataset into memory. This is how you build tools that can handle "enterprise-scale" data without needing an enterprise-scale server.

📋 Practical Task

Exercise: Building a Time-Filtered Log Analyzer

Your task is to extend the data processor we just built. Instead of just counting all errors, you need to modify the ProcessErrorReport logic to only include errors that occurred within a specific time window.

Requirements:

• Modify the method signature to accept two additional parameters: DateTime startDate and DateTime endDate.
• Add a .Where() clause to the LINQ chain that filters the LogEntry timestamps to fall between these two dates (inclusive).
• Update the output report header to include the date range being analyzed (e.g., "Service Error Report from 2023-01-01 to 2023-01-07").
• Ensure the program still uses the streaming IEnumerable approach to maintain memory efficiency.