I’ve got a folder on my desktop that's basically a digital junk drawer—hundreds of screenshots, downloaded PDFs, and "final_version_v2" documents. I suspect there are dozens of duplicates taking up space, but scrolling through them manually is a nightmare. Let's build something to find them. My goal is simple: find every file that has the exact same content, regardless of what the file is named.

The "Same Name" Trap

My first instinct was to just look for files with the same name. I mean, if I have report.pdf and report(1).pdf, they're probably the same, right? Let's try a quick prototype.

import os

files = os.listdir('.')
name_counts = {}

for f in files:
    name_counts[f] = name_counts.get(f, 0) + 1

duplicates = [name for name, count in name_counts.items() if count > 1]
print(f"Duplicates: {duplicates}")

This is useless. In most operating systems, you can't even have two files with the exact same name in the same folder. And if they're in different folders, the names might be different even if the content is identical. I need a way to "fingerprint" the actual data inside the file.

Dealing with the Memory Hog

Okay, if names don't work, I'll just read the files and compare the bytes. I'll use a dictionary where the key is the file content and the value is a list of paths. Seems straightforward.

import os

hashes = {}
for root, dirs, files in os.walk('.'):
    for f in files:
        path = os.path.join(root, f)
        with open(path, 'rb') as file:
            content = file.read() # Read the whole thing
            if content in hashes:
                hashes[content].append(path)
            else:
                hashes[content] = [path]

I tried running this on my "Downloads" folder, and my laptop's fans started screaming. Then Python crashed with a MemoryError. Why? Because I tried to read a 2GB ISO file directly into a variable. You can't just slam entire files into RAM and expect it to work. I need a hash—a short, unique string that represents the file's contents without needing the contents themselves to stay in memory.

Refining the Fingerprint

I'll use hashlib to create a SHA-256 hash. But I still can't read() the whole file. Instead, I'll read the file in small chunks—say, 4KB at a time—and update the hash incrementally. This keeps the memory footprint tiny regardless of whether the file is 1KB or 10GB.

import hashlib
import os

def get_file_hash(path):
    sha256 = hashlib.sha256()
    with open(path, 'rb') as f:
        while chunk := f.read(4096):
            sha256.update(chunk)
    return sha256.hexdigest()

# Now we can map hashes to paths
hash_map = {}
for root, dirs, files in os.walk('.'):
    for f in files:
        path = os.path.join(root, f)
        file_hash = get_file_hash(path)
        hash_map.setdefault(file_hash, []).append(path)

This works, and it's stable. But as I watched it run, I realized it was still slow. It's calculating a heavy SHA-256 hash for every single file, even if one file is 10 bytes and another is 10 gigabytes. There's no way those are duplicates.

The Optimization Shortcut

If two files are duplicates, they must have the exact same size. This is a massive shortcut. I can group files by size first. If a size is unique, I don't even need to open the file or calculate a hash. I only hash files that share the same byte count.

Here is the final logic I landed on: first, group by size; second, for any size group with more than one file, calculate the hashes to confirm they are actually identical.

import os
import hashlib
from collections import defaultdict

def find_duplicates(target_dir):
    size_map = defaultdict(list)
    
    # Phase 1: Group by size (fast)
    for root, _, files in os.walk(target_dir):
        for f in files:
            path = os.path.join(root, f)
            try:
                size_map[os.path.getsize(path)].append(path)
            except OSError:
                continue # Skip files we can't access

    duplicates = []
    
    # Phase 2: Hash only the candidates (precise)
    for size, paths in size_map.items():
        if len(paths) < 2:
            continue
            
        hash_map = defaultdict(list)
        for path in paths:
            # We'll use the chunking method from before
            file_hash = get_file_hash(path) 
            hash_map[file_hash].append(path)
            
        for h, p_list in hash_map.items():
            if len(p_list) > 1:
                duplicates.append(p_list)
                
    return duplicates

By adding that size-check layer, I reduced the number of files my script had to actually "read" by about 80%. It's the difference between a script that feels clunky and one that feels professional.

📋 Practical Task

Exercise: Building a Storage-Saving Duplicate Reporter

Now it's your turn. Expand on the logic we explored to build a tool that doesn't just find duplicates, but calculates how much space you would save by deleting them.

Write a program that does the following:

• Scans a directory of your choice for duplicate files (using the size-then-hash approach).
• For every set of duplicates found, identify one "original" and treat the others as "waste".
• Calculate the total sum of the file sizes of all "waste" files.
• Print a report showing:
  1. The groups of duplicate files found.
  2. The total amount of disk space (in MB or GB) that could be reclaimed.

Constraint: Ensure your program handles PermissionError or FileNotFoundError gracefully using a try/except block inside the file-walking loop so it doesn't crash when encountering system-protected files.