How do I handle large files without crashing my memory?

This is the first mistake I see people make. It's tempting to just use os.ReadFile(), dump the whole file into a byte slice, and pass that to the hasher. That works great for a few small text files, but the moment you hit a 4GB video file, your program is going to panic with an out-of-memory error.

The right way to do this in Go is to stream the data. The hash.Hash interface in the crypto package implements io.Writer. This means you can use io.Copy() to pipe the file content directly into the hash function. Go will handle the buffering under the hood, meaning you only ever have a few kilobytes in memory regardless of whether the file is 1MB or 100GB.

func calculateHash(filePath string) (string, error) {
    f, err := os.Open(filePath)
    if err != nil {
        return "", err
    }
    defer f.Close()

    h := sha256.New()
    // This is the magic part: io.Copy streams from the file to the hash
    if _, err := io.Copy(h, f); err != nil {
        return "", err
    }

    return fmt.Sprintf("%x", h.Sum(nil)), nil
}

Which hashing algorithm should I actually use?

For a deduplicator, you're balancing speed against the risk of "collisions" (where two different files produce the same hash). While MD5 is faster, it's cryptographically broken and has a higher collision rate. I generally stick with SHA-256. It's more than fast enough for local disk I/O, and the odds of a collision are astronomically low—low enough that you can practically treat the hash as a unique fingerprint for the file.

One little tip: always return the hash as a hex-encoded string (using %x in fmt.Sprintf or hex.EncodeToString). It's much easier to store and compare as a map key than dealing with raw byte slices.

How do I organize the map to keep track of the duplicates?

You don't want a simple map[string]bool because that only tells you if a file has been seen before; it doesn't tell you which files are the duplicates. Instead, use a map where the key is the hash and the value is a slice of strings containing the file paths.

I find it's cleanest to append every file path to the slice for its corresponding hash. Once you've scanned the whole directory, you just iterate through the map and ignore any entries where the slice length is exactly 1. If the length is 2 or more, you've found your duplicates.

filesByHash := make(map[string][]string)

for _, path := range allFiles {
    hash, err := calculateHash(path)
    if err != nil {
        continue // In a real app, you'd probably log this
    }
    filesByHash[hash] = append(filesByHash[hash], path)
}

for hash, paths := range filesByHash {
    if len(paths) > 1 {
        fmt.Printf("Duplicate group [%s]: %v\n", hash, paths)
    }
}

📋 Practical Task

Build a Recursive Directory Deduplicator

Your task is to create a command-line tool that finds duplicate files within a specified directory and all of its sub-directories.

• The program should take a directory path as a command-line argument.
• Use path/filepath.WalkDir to recursively find all files in the directory tree.
• Implement the streaming SHA-256 hashing logic to ensure the program can handle files of any size.
• Store the results in a map to group files by their hashes.
• Print the results to the console in a clear format: for every set of duplicates found, print the hash followed by the list of file paths that share that hash.
• Ensure you handle file permission errors gracefully (e.g., skip files that cannot be opened instead of crashing the whole program).