Hashing is one of those things that seems straightforward until you have to deal with data that doesn't fit comfortably in RAM. I've seen plenty of developers treat the crypto package like a black box where you throw in a byte slice and get a checksum back. That works fine for a short password or a small JSON blob, but when you're dealing with something like a 2GB disk image or a large log file, that approach will crash your app with an out-of-memory error faster than you can say "segmentation fault."

The temptation to load everything at once

When you first look at the crypto/sha256 package, you'll see a function called Sum256. It's incredibly convenient. You pass it a slice of bytes, and it returns a fixed-size array. It looks like this:

// The "quick and dirty" way (dangerous for large files)
data, _ := os.ReadFile("huge_backup.zip") 
hash := sha256.Sum256(data)
fmt.Printf("%x", hash)

The problem here is os.ReadFile. It pulls the entire file into memory. If your file is larger than your available heap, you're done. Even if it fits, you're putting unnecessary pressure on the garbage collector. In a production environment, this is a ticking time bomb. You don't want your checksum utility to be the reason the server swaps to disk and grinds to a halt.

Streaming for constant memory usage

The professional way to handle this—and how Go's crypto packages are actually designed to be used—is through the hash.Hash interface. Instead of a one-shot function, you create a hash object, feed it data in chunks, and then finalize the result. This allows you to process a file of any size using a tiny, constant amount of memory.

I usually pair sha256.New() with io.Copy. Since a file is an io.Reader and the hash object implements io.Writer, you can pipe the data directly from the disk into the hashing engine without ever holding more than a few kilobytes in memory at once:

f, _ := os.Open("huge_backup.zip")
defer f.Close()

h := sha256.New()
if _, err := io.Copy(h, f); err != nil {
    log.Fatal(err)
}

// h.Sum(nil) appends the current hash to the provided slice
checksum := h.Sum(nil)
fmt.Printf("%x", checksum)

Notice how h.Sum(nil) works. It doesn't take the data to be hashed; it just tells the engine, "I'm done feeding you, give me the final result." By passing nil, you're just asking for the hash itself as a new slice.

Choosing between MD5 and SHA256

You'll notice that crypto/md5 has an almost identical API to sha256. You might be wondering why we have both. Here is the rule of thumb I use: if you are doing anything related to security, passwords, or digital signatures, do not use MD5. It's cryptographically broken; collisions are trivial to generate now. If an attacker can create two different files with the same MD5 hash, they can swap a legitimate binary for a malicious one without you noticing.

However, MD5 is significantly faster than SHA256. If you're building a cache system where you just need to detect if a file has changed—and you aren't worried about a malicious actor intentionally trying to trick the system—MD5 is still a viable tool. But honestly? With modern CPU instructions (like Intel's SHA extensions), the performance gap has narrowed. Unless you're hashing terabytes of data per second, just stick with SHA256. It's better to be secure by default than to save a few CPU cycles and regret it later.

📋 Practical Task

Exercise: Building a File Integrity Verifier

Write a program that verifies if a file has been tampered with by comparing its current SHA256 checksum against a known "golden" hash provided as a command-line argument.

• The program should accept two arguments: the path to a file and the expected hex-encoded SHA256 string.
• You must use a streaming approach (io.Copy) to ensure the program can handle files larger than the available system memory.
• The program should print "Checksum verified" if they match, or "Checksum mismatch!" if they differ.
• Use fmt.Sprintf("%x", ...) or hex.EncodeToString to compare the resulting bytes with the input string.