A few years ago, I was reviewing a PR from a junior engineer who was building a log analysis tool. He had used os.ReadFile() to pull a 2GB production log file into a byte slice, then split that slice by newlines. It worked perfectly on his laptop with a small sample file, but the moment it hit the staging server with real data, the pod crashed instantly with an Out-of-Memory (OOM) error. He was baffled because "the code was logically correct." The problem wasn't the logic; it was the strategy. You can't treat a massive stream of data like a small string in memory.

This is where the bufio package comes in. Instead of gulping down the entire file at once, bufio lets you sip it. It creates a buffer in memory—a small "waiting room"—that reads chunks of the file from the disk and serves them to you piece by piece. This keeps your memory footprint tiny regardless of whether the file is 10 KB or 10 TB.

Streaming lines with bufio.Scanner

For 90% of your tasks—like reading a config file or parsing logs line-by-line—bufio.Scanner is your best friend. It abstracts away the complexity of looking for newline characters and managing the buffer for you. You just tell it to Scan(), and it gives you the next token.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("server.log")
    if err != nil {
        fmt.Println("Error opening file:", err)
        return
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        // scanner.Text() returns the current line as a string
        line := scanner.Text()
        fmt.Println("Processing line:", line)
    }

    if err := scanner.Err(); err != nil {
        fmt.Println("Error reading file:", err)
    }
}

One thing I want you to watch out for: Scanner has a default maximum token size (usually 64KB). If you encounter a line longer than that—which happens more often than you'd think in JSON logs—Scan() will return false and scanner.Err() will tell you the line was too long. If you expect massive lines, you'll need to use a bufio.Reader or manually adjust the buffer via scanner.Buffer().

Taking granular control with bufio.Reader

Sometimes Scanner is too simplistic. Maybe you need to read until a specific character that isn't a newline, or you need to "peek" at the next few bytes without actually consuming them from the stream. That's when you reach for bufio.Reader.

Unlike the scanner, the reader doesn't just give you "the next line." It gives you methods like ReadString, ReadBytes, and Peek. It's more verbose, but it's far more powerful for implementing custom protocols or parsing complex file formats.

package main

import (
    "bufio"
    "fmt"
    "os"
    "strings"
)

func main() {
    // Using strings.NewReader to simulate a file for this example
    input := "User: admin\nPass: 12345\nLog: Success\n"
    reader := bufio.NewReader(strings.NewReader(input))

    for {
        // Read until the newline character
        line, err := reader.ReadString('\n')
        if err != nil {
            break // End of file
        }
        fmt.Print("Reader found: ", line)
    }
}

I generally suggest starting with Scanner because the code is cleaner. Only switch to Reader when you find yourself fighting the Scanner's limitations. Most of the time, the simplicity of a for scanner.Scan() loop is exactly what you want.

📋 Practical Task

Building a Memory-Efficient Log Error Filter

Create a program that reads a large text file named system.log. Instead of loading the whole file into memory, use bufio.Scanner to read the file line-by-line. Your program should only print lines that contain the word "ERROR" or "CRITICAL".

Requirements:

• The program must handle the file opening and closing correctly using defer.
• It must use a bufio.Scanner to ensure memory usage remains constant regardless of file size.
• It must check scanner.Err() after the loop to report any read errors.
• If the file is not found, it should print a helpful error message and exit gracefully.