How do I just read a whole file into memory quickly?

If you're dealing with a small configuration file or a simple text document, don't overcomplicate things. You don't need a buffer or a complex loop. The os package gives us ReadFile, which does exactly what it sounds like: it opens the file, reads the entire thing into a byte slice, and closes the file for you in one shot.

package main

import (
	"fmt"
	"os"
)

func main() {
	// Let's assume we have a 'config.txt' file
	data, err := os.ReadFile("config.txt")
	if err != nil {
		fmt.Println("Error reading file:", err)
		return
	}

	// data is a []byte, so we cast it to a string to print it
	fmt.Println(string(data))
}

It's convenient, but be careful. If you try to use this on a 10GB server log, your program will crash because it tries to shove that entire 10GB into your RAM. That's where we need a different approach.

What's the best way to process a massive file line-by-line?

When the file is too big for memory, we use bufio.Scanner. Think of it like a sliding window that reads the file in small chunks. This is the bread and butter of log processing. You open the file with os.Open, wrap it in a scanner, and iterate through it.

package main

import (
	"bufio"
	"fmt"
	"log"
	"os"
	"strings"
)

func main() {
	file, err := os.Open("server.log")
	if err != nil {
		log.Fatal(err)
	}
	// I always put the Close() right after the error check. 
	// It ensures the file descriptor is released even if the function panics.
	defer file.Close()

	scanner := bufio.NewScanner(file)
	for scanner.Scan() {
		line := scanner.Text()
		// Let's only print lines that contain "ERROR"
		if strings.Contains(line, "ERROR") {
			fmt.Println("Found an issue:", line)
		}
	}

	if err := scanner.Err(); err != nil {
		log.Fatal(err)
	}
}

One thing to watch out for: scanner.Scan() has a default buffer size. If you have a single line of text that is ridiculously long (like a giant JSON blob on one line), the scanner will fail. If that happens, you'll need to use reader.ReadLine() or increase the buffer manually, but for 95% of use cases, the scanner is your best bet.

How do I write to a file without overwriting everything?

Writing is where people usually get tripped up because os.WriteFile behaves like ReadFile—it handles everything at once, but it truncates (wipes) the file if it already exists. If you're building a logger, you want to append to the file.

To do this, you use os.OpenFile with specific flags. I usually pair this with bufio.Writer because writing to a disk is slow. Buffering collects your data in memory and writes it in larger chunks, which is significantly more performant.

package main

import (
	"bufio"
	"fmt"
	"os"
)

func main() {
	// O_APPEND: Add to the end of the file
	// O_CREATE: Create the file if it doesn't exist
	// O_WRONLY: Open for writing only
	file, err := os.OpenFile("app.log", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer file.Close()

	writer := bufio.NewWriter(file)
	
	_, err = writer.WriteString("User logged in: admin\n")
	if err != nil {
		fmt.Println(err)
	}

	// CRITICAL: bufio.Writer stores data in a buffer. 
	// If you don't call Flush(), your last few lines might never actually hit the disk.
	writer.Flush()
}

That Flush() call is the most common bug I see in junior Go code. If your program finishes and your log file is empty, 9 times out of 10, you forgot to flush the buffer.

📋 Practical Task

Build a Log Filtering Tool

Create a program that reads a file named input.log and writes all lines containing the word "CRITICAL" into a new file named alerts.txt.

• The program should use bufio.Scanner to read the input file to ensure it can handle files of any size.
• The program should use bufio.Writer to write the filtered lines to the output file.
• Ensure that you properly close both files using defer and that you call Flush() on your writer before the program exits.
• Handle errors for opening and creating the files gracefully.