When you're dealing with data streams in Go, you'll eventually hit a crossroads: do you want the whole thing in memory right now, or do you just want to move it from point A to point B? This is exactly what io.ReadAll and io.Copy are for. If you pick the wrong one, you're either writing more code than you need or, worse, you're writing a program that crashes the moment a user uploads a 5GB file.

To see this in action, I'm going to build a tiny "Backup Utility." The goal is simple: read a small config.txt file to see if backups are enabled, and if they are, stream a large system.log file into a backup.log file.

Grabbing the config with io.ReadAll

First, let's handle the configuration. Since I know config.txt is just a few lines of text, I don't need to worry about memory. I just want the bytes in a slice so I can convert them to a string and check the contents.

func checkConfig() (bool, error) {
    f, err := os.Open("config.txt")
    if err != nil {
        return false, err
    }
    defer f.Close()

    // io.ReadAll reads until EOF and returns a []byte
    data, err := io.ReadAll(f)
    if err != nil {
        return false, err
    }

    return string(data) == "BACKUP_ENABLED=true", nil
}

io.ReadAll is great here. It's concise. It handles the loop of reading chunks for you and grows the slice as needed. If the file is small, this is the most ergonomic way to get the job done.

The mistake: Trying to ReadAll the logs

Now, let's move the actual log file. My first instinct—which is a habit I had to break early in my Go career—was to just use io.ReadAll again because it's so easy. I thought, "I'll just read the log file into a variable and then write that variable to the backup file."

// DON'T DO THIS for large files
data, err := io.ReadAll(logFile) 
if err != nil {
    return err
}
os.WriteFile("backup.log", data, 0644)

Here is the problem: io.ReadAll allocates a byte slice. If system.log is 10MB, no big deal. If it's 10GB, your program will try to allocate 10GB of RAM, the OS will panic, and your process will be killed with an "out of memory" (OOM) error. You should almost never use io.ReadAll on a file unless you have a strict guarantee about its maximum size.

Streaming the data with io.Copy

Instead of pulling the data into our program's memory, we can tell Go to just "pipe" it. io.Copy takes a Writer (the destination) and a Reader (the source) and moves data between them using a small, fixed-size buffer internally.

Here is how I'll actually implement the backup logic:

func performBackup() error {
    source, err := os.Open("system.log")
    if err != nil {
        return err
    }
    defer source.Close()

    destination, err := os.Create("backup.log")
    if err != nil {
        return err
    }
    defer destination.Close()

    // io.Copy streams from source to destination.
    // It doesn't load the whole file into RAM.
    written, err := io.Copy(destination, source)
    if err != nil {
        return err
    }

    fmt.Printf("Successfully backed up %d bytes\n", written)
    return nil
}

Notice the difference. I'm not creating a []byte slice. The data flows through a small window (usually 32KB) and goes straight to the disk. This means whether the log file is 1KB or 1TB, the memory usage of my program remains constant and tiny.

Choosing your tool

To wrap this up, here is my rule of thumb when I'm reviewing code:

• Use io.ReadAll when you specifically need the entire content as a slice to manipulate it, and you know for a fact the input is small.
• Use io.Copy when you are moving data from one place to another. It is safer, faster, and significantly more memory-efficient.

📋 Practical Task

Exercise: Build a Log Filter Streamer

Create a program that simulates a log filtering tool. Your program should:

1. Read a small file named filter.txt using io.ReadAll to get a keyword (e.g., "ERROR").
2. Open a larger file named app.log.
3. Instead of copying the whole file, use a bufio.Scanner to read the log file line-by-line.
4. If a line contains the keyword from filter.txt, write that line to a new file called filtered.log.
5. Bonus: Use io.Copy to copy the rest of the app.log to a full_archive.log file as a secondary backup step.

This will test your ability to distinguish between when to load a whole file into memory and when to stream data to avoid memory exhaustion.