A few years ago, I worked with a colleague who was managing a massive archive of legacy project assets. He had this habit of "saving versions" by copying folders, and after three years, he had about 400GB of data that he suspected was mostly redundant. He tried to write a quick script to find duplicates by comparing filenames, but that failed miserably because he'd renamed things like logo_final.png, logo_final_v2.png, and logo_final_ACTUAL_FINAL.png. They were identical files, but the names were different. He spent a whole weekend manually deleting files before he realized he needed to look at the actual bytes, not the labels.

When you're building a deduplication tool, you have to ignore the metadata and focus on the content. The most efficient way to do this in Java is by creating a "fingerprint" of the file using a cryptographic hash function. Instead of comparing every byte of one file against every byte of every other file—which would be an O(n²) nightmare—you calculate a unique hash for each file and store it in a map. If two files produce the same hash, you've found a duplicate.

Fingerprinting Files with MessageDigest

To generate these fingerprints, we use the java.security.MessageDigest class. While you'll hear a lot about SHA-256 being the standard for security, for a local deduplication tool, MD5 is often sufficient and slightly faster. I usually lean towards SHA-256 anyway just to avoid the tiny possibility of a collision, but the implementation is the same.

public String calculateHash(Path path) throws IOException, NoSuchAlgorithmException {
    MessageDigest digest = MessageDigest.getInstance("SHA-256");
    try (InputStream is = Files.newInputStream(path)) {
        byte[] buffer = new byte[8192];
        int read;
        while ((read = is.read(buffer)) != -1) {
            digest.update(buffer, 0, read);
        }
    }
    
    // Convert the byte array to a hex string for easy mapping
    StringBuilder sb = new StringBuilder();
    for (byte b : digest.digest()) {
        sb.append(String.format("%02x", b));
    }
    return sb.toString();
}

Notice that I'm reading the file in chunks. If you try to read a 2GB video file into a byte array all at once, you're going to hit an OutOfMemoryError. Streaming the file into the digest allows you to process files of any size with a constant memory footprint.

Grouping Files by Content Hash

Once you can generate a hash, the logic becomes a matter of data organization. You want a Map where the key is the hash string and the value is a List of paths that produced that hash. If a list contains more than one path, you've found your duplicates.

I recommend using Files.walk to traverse the directory tree. It returns a Stream<Path>, which fits perfectly with Java's functional style. You can filter out directories (since we only care about files) and then collect the results into your map.

Map<String, List<Path>> hashes = Files.walk(startPath)
    .filter(Files::isRegularFile)
    .collect(Collectors.groupingBy(path -> {
        try {
            return calculateHash(path);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }));

One thing to keep in mind: I'm wrapping the checked exceptions in a RuntimeException inside the lambda. In a production environment, you'd probably want a more robust way to handle I/O errors—like skipping a file that the OS refuses to let you read—rather than crashing the whole stream.

Optimizing the Scan Process

Hashing every single file can be slow if you have tens of thousands of them. A pro tip to speed this up is to filter by file size first. Two files cannot be identical if they have different sizes. You can create an initial map of Long (size) > List<Path>. Only for the size-groups that have more than one file do you actually bother calculating the SHA-256 hash.

This "cheap check" before the "expensive check" is a common pattern in software engineering. It saves a massive amount of Disk I/O because you aren't reading the contents of every unique file on your hard drive; you're only reading the ones that are already suspicious.

📋 Practical Task

The Duplicate Asset Purge Tool

Your task is to build a command-line utility that identifies duplicate files within a given directory. The tool should follow these requirements:

• Accept a directory path as a command-line argument.
• Recursively scan the directory and all its subdirectories.
• Use SHA-256 to determine if files are identical.
• Print only the groups of files that are duplicates. Each group should be clearly separated, listing the full path of every file that shares the same hash.
• Ensure the program can handle large files without crashing (use a buffer/stream).

Bonus Challenge: Implement the "size-first" optimization mentioned in the lesson to avoid hashing files that have unique byte counts.