A few years ago, I was helping a colleague clean up a shared staging server that had become a graveyard of "temporary" backups. He had folders named backup_final, backup_final_v2, and backup_actually_final_this_time. He was convinced he had 200GB of unique data, but the disk was screaming for space. We spent an hour manually comparing folder sizes before we realized that about 70% of the files were exact binary copies of each other. That's when we sat down and wrote a small Go utility to find the duplicates. It's a classic problem, but it's the perfect way to practice combining I/O operations with Go's map structures.

Quick Wins with File Sizes

If you're building a deduplication tool, the biggest mistake you can make is hashing every single file immediately. If you have a 10GB ISO file and a 1KB text file, they obviously aren't duplicates. Why waste CPU cycles calculating a SHA-256 checksum for a 10GB file just to find out it's the only file of that size on the disk? I always tell people to start with the "cheap" metadata first.

In Go, we can use os.ReadDir to traverse a directory and os.Stat (or the FileInfo returned by the directory read) to get the file size. The most efficient way to group these is using a map where the key is the file size (an int64) and the value is a slice of strings containing the paths to the files. If a size category only contains one file, you can safely ignore it and move on.

type FileGroup map[int64][]string

func groupBySize(root string) (FileGroup, error) {
    groups := make(FileGroup)
    files, err := os.ReadDir(root)
    if err != nil {
        return nil, err
    }

    for _, f := range files {
        if f.IsDir() {
            continue
        }
        info, err := f.Info()
        if err != nil {
            continue
        }
        size := info.Size()
        groups[size] = append(groups[size], f.Name())
    }
    return groups, nil
}

The Heavy Lifting with SHA-256

Once you've narrowed the field down to files that share the exact same byte count, it's time to get surgical. Now we need to verify if the content is actually identical. I prefer crypto/sha256 for this because the collision probability is infinitesimally low for this kind of tool.

One detail that often trips people up is loading the entire file into memory with os.ReadFile. If you hit a 4GB file, your program will likely crash with an out-of-memory error. Instead, use io.Copy to stream the file content directly into the hasher. This keeps your memory footprint tiny regardless of the file size.

func calculateHash(path string) (string, error) {
    f, err := os.Open(path)
    if err != nil {
        return "", err
    }
    defer f.Close()

    h := sha256.New()
    if _, err := io.Copy(h, f); err != nil {
        return "", err
    }

    return fmt.Sprintf("%x", h.Sum(nil)), nil
}

Identifying the Culprits

Now we bring it all together. You iterate through your size-grouped map, and for every group that has more than one file, you create a new map for hashes. The key is the SHA-256 string, and the value is again a slice of paths.

I like to think of this as a two-stage filter. Stage one (Size) removes the obvious non-matches. Stage two (Hash) confirms the duplicates. At the end of the process, any hash group with a slice length greater than one is a set of duplicates. You can then decide whether to just print them to the console or automatically delete the redundant copies—though I'd strongly advise printing them first unless you're feeling particularly brave with your os.Remove calls.

📋 Practical Task

Build the "Disk Space Recovery" Scanner

Create a command-line tool that accepts a directory path as an argument. The program should scan the directory and identify all duplicate files.

• The tool must first group files by size.
• It must only calculate SHA-256 hashes for files that share the same size.
• The output should be formatted to show the hash of the duplicate set, followed by the list of file paths that share that hash.
• Ensure you use io.Copy to handle large files without crashing your memory.

Test Case: Create a folder with three files: a.txt (content: "hello"), b.txt (content: "hello"), and c.txt (content: "world"). Your tool should identify a.txt and b.txt as duplicates and ignore c.txt.