Why does my app crash with an OutOfMemoryError when I try to read a large log file?

I've seen this happen to almost every developer at some point. You probably used Files.readAllLines() or Files.readAllBytes(). These methods are convenient for small config files, but they are dangerous for anything substantial. The problem is that they attempt to load the entire file into your JVM's heap memory at once.

If you're trying to read a 2GB server log on a machine with a 1GB heap limit, you're going to crash. It doesn't matter how fast your SSD is; you've simply run out of room to put the data. To handle large files, you have to shift your mindset from "loading" the file to "streaming" the file.

What's the best way to process a file line-by-line without killing my RAM?

In modern Java, Files.lines() is your best friend. It returns a Stream<String>, which reads the file lazily. It doesn't pull the whole file into memory; it reads one line, lets you process it, and then moves to the next. It's incredibly memory-efficient.

Let's say we have a massive access.log and we only care about lines that contain a "404" error. Here is how I'd write that:

import java.nio.file.*;
import java.io.IOException;
import java.util.stream.Stream;

public class LogAnalyzer {
    public void findErrors(String filePath) {
        Path path = Paths.get(filePath);
        
        // We use try-with-resources because the stream holds an open file handle
        try (Stream<String> lines = Files.lines(path)) {
            lines.filter(line -> line.contains(" 404 "))
                 .forEach(line -> System.out.println("Found missing page: " + line));
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Notice the try-with-resources block. A common mistake is forgetting that Files.lines() opens a file handle. If you don't close it, you'll eventually hit the OS limit for open files, and your app will start throwing IOExceptions for no apparent reason.

Is there an even faster way if I'm doing heavy random access or massive reads?

If Files.lines() is still too slow—perhaps because you're doing complex binary parsing or you need to jump around the file—you need to look at MappedByteBuffer via FileChannel. This is called "memory-mapping."

Essentially, it tells the operating system to map a region of the file directly into virtual memory. Instead of copying data from the kernel space to the user space (which is what standard I/O does), the JVM accesses the file as if it were an array in memory. It's a bit more "low-level," but for multi-gigabyte files where performance is the only metric that matters, it's the gold standard.

I wouldn't use this for simple text processing—it's overkill. But if you're building a database engine or a high-frequency trading log parser, that's where you'll spend your time. Just be careful: MappedByteBuffer is powerful, but it can be tricky to "unmap" manually in Java, often relying on the garbage collector to clean up the mapping.

📋 Practical Task

Exercise: Large-Scale Log Error Counter

You are tasked with analyzing a simulated 5GB production log file. Your goal is to count the total number of occurrences of the word "CRITICAL" across the entire file without exceeding a 256MB JVM heap limit.

• Create a class called LogErrorCounter.
• Implement a method long countCriticalErrors(String path) that uses Files.lines() to scan the file.
• The method should return the total count of lines containing the string "CRITICAL".
• Ensure that you are using try-with-resources to prevent memory leaks.
• Test your solution with a large file (you can generate one using a simple loop that writes "INFO" and "CRITICAL" lines to a file) to verify that the memory usage remains constant regardless of file size.