When you first start working with the file system in Node.js, you'll likely reach for fs.readFile(). It's intuitive. You tell Node to get the file, and it hands you the whole thing in a callback or a promise. For a config file or a small JSON manifest, this is perfectly fine. But here is where most developers hit a wall: they assume that the only constraint when reading a file is the time it takes to read it.

The "Read it All at Once" Fallacy

The misconception is that fs.readFile() is simply a "slower" version of a stream for large files. In reality, the problem isn't speed—it's memory. When you use readFile(), Node.js must allocate a buffer large enough to hold the entire file in RAM before it can pass it to your code.

Imagine you're building a tool to analyze a 4GB server log file, but your container is limited to 2GB of RAM. If you try to use fs.readFile(), your process won't just be slow; it will crash with an ERR_STRING_TOO_LONG or a heap out-of-memory error. You're trying to cram a gallon of water into a pint glass. I've seen this happen in production environments more times than I'd like to admit, usually during a "quick" migration script that worked fine on a developer's laptop but died in the cloud.

Thinking in Chunks and Pipes

This is where Streams come in. Instead of treating a file as one giant block of data, a Stream treats it as a continuous flow of small pieces called "chunks." Using fs.createReadStream(), Node reads a small slice of the file (usually 64KB by default), processes it, and then moves to the next slice. Your memory usage stays flat, whether the file is 10MB or 10TB.

const fs = require('fs');

// Instead of reading the whole file into memory...
const readStream = fs.createReadStream('massive-log.txt', { encoding: 'utf8' });

readStream.on('data', (chunk) => {
  console.log('Received a new chunk of data:');
  console.log(chunk); // This is just a piece of the file
});

readStream.on('end', () => {
  console.log('Finished reading the entire file.');
});

While the .on('data') approach works, it's a bit low-level. In the real world, we usually want to move data from a source (a readable stream) to a destination (a writable stream). This is where .pipe() becomes your best friend. It connects the two streams together, automatically managing the flow of data.

const fs = require('fs');

const src = fs.createReadStream('huge-video.mp4');
const dest = fs.createWriteStream('backup-video.mp4');

// This reads from src and writes to dest efficiently
src.pipe(dest);

The Hidden Danger: Backpressure

Now, here is a nuance that separates junior devs from senior engineers: backpressure. What happens if your readable stream is reading data at 100MB/s, but your writable stream (maybe a slow network upload) can only write at 10MB/s? If you manually handle 'data' events, the data will pile up in your RAM, and you're right back to the memory crash we started with.

The .pipe() method handles this for you. It monitors the internal buffer of the destination stream. If the destination says, "Wait, I'm full!", .pipe() tells the source stream to pause reading until the destination catches up. I highly recommend using .pipe() or the newer pipeline() utility from the stream module rather than manually listening to events, simply because managing backpressure by hand is a tedious exercise in bug-hunting.

📋 Practical Task

The Log File Error Filter Stream

You have a massive log file named server.log. You need to create a script that reads this file and writes only the lines containing the word "ERROR" into a new file called errors.log.

Requirements:

• Do NOT use fs.readFile() or fs.readFileSync().
• Use fs.createReadStream and fs.createWriteStream.
• Use a Transform stream (from the stream module) to filter the data. The transform stream should check if a chunk contains "ERROR" and only pass those parts through to the writable stream.
• Use pipeline from the stream module to connect the read stream, the transform stream, and the write stream, ensuring proper error handling and backpressure management.