How do I actually identify a "duplicate" when the data is messy?

In a perfect world, you'd just compare two strings and be done. But in the real world, you'll have "John Doe" in one record and " john doe " in another. If you compare those directly, Python will tell you they're different, and your tool will fail.

The trick is normalization. Before you even think about comparing records, you need to strip the noise. I usually create a "normalization key" for each record. This is a version of the data that is stripped of whitespace, converted to lowercase, and has special characters removed.

def get_norm_key(contact):
    # We'll use email as the primary unique identifier
    email = contact.get('email', '').strip().lower()
    return email

contacts = [
    {"name": "Alice Smith", "email": "ALICE@example.com"},
    {"name": "alice smith", "email": " alice@example.com "},
    {"name": "Bob Jones", "email": "bob@test.com"}
]

# Instead of comparing the whole dict, we compare the normalized keys
keys = [get_norm_key(c) for c in contacts]
# keys becomes ['alice@example.com', 'alice@example.com', 'bob@test.com']

By reducing the data to its simplest form, you eliminate the most common reasons why deduplication fails.

Is there a way to find duplicates without using nested loops?

If you use a nested loop (comparing every contact to every other contact), your program's performance will tank as your list grows. That's O(n²) complexity, and it's a classic rookie mistake. I've seen scripts that take ten minutes to run on 5,000 contacts because of this.

Instead, use a dictionary to group your contacts. The key is your normalized identifier, and the value is a list of all records that match that identifier. This lets you find every duplicate in a single pass through the data.

from collections import defaultdict

def group_duplicates(contacts):
    groups = defaultdict(list)
    for contact in contacts:
        key = get_norm_key(contact)
        if key: # Ignore records without an email
            groups[key].append(contact)
    return groups

# Now you can easily see who has more than one entry
grouped = group_duplicates(contacts)
duplicates = {k: v for k, v in grouped.items() if len(v) > 1}

This approach is significantly faster because dictionary lookups in Python are essentially instant.

Once I've found the duplicates, how do I merge them into one "golden" record?

This is where it gets subjective. You have to decide which data "wins." For example, if one record has a phone number and the other doesn't, you obviously want the phone number in your final result. I prefer a "filling the gaps" strategy.

I typically start with the first record in the duplicate group and then iterate through the others, updating the main record only if the current value is empty or null. It's a simple way to ensure you aren't losing information during the merge.

def merge_contacts(records):
    if not records:
        return None
    
    # Start with the first record as the base
    merged = records[0].copy()
    
    for record in records[1:]:
        for key, value in record.items():
            # If the merged record is missing data, or has an empty string, fill it
            if not merged.get(key) and value:
                merged[key] = value
                
    return merged

# Example: Alice has a phone in one record and a city in another
alice_dupes = [
    {"name": "Alice Smith", "email": "alice@example.com", "phone": "555-1234", "city": ""},
    {"name": "Alice Smith", "email": "alice@example.com", "phone": "", "city": "New York"}
]

final_alice = merge_contacts(alice_dupes)
# Result: {'name': 'Alice Smith', 'email': 'alice@example.com', 'phone': '555-1234', 'city': 'New York'}

Just be careful: if two records have different phone numbers, this logic keeps the first one it found. Depending on your needs, you might want to flag those for manual review instead of blindly merging.

📋 Practical Task

Exercise: The Corporate Directory Cleanup Script

You've been handed a messy list of employee contacts from three different legacy systems. Your task is to build a cleanup script that merges these records into a single, unique list.

Requirements:

• Create a function that normalizes the email address (lowercase and stripped of whitespace).
• Identify all duplicates based on that normalized email.
• Merge the duplicates so that the final record contains the most complete information (i.e., if one record has a department and another doesn't, the final record should have the department).
• Return a list of unique, merged contact dictionaries.

raw_contacts = [
    {"name": "Jim Halpert", "email": "jim@dundermifflin.com", "dept": "Sales", "phone": "555-0101"},
    {"name": "jim halpert", "email": " JIM@dundermifflin.com ", "dept": "", "phone": "555-0101"},
    {"name": "Pam Beesly", "email": "pam@dundermifflin.com", "dept": "Reception", "phone": ""},
    {"name": "Pam Beesly", "email": "pam@dundermifflin.com", "dept": "Reception", "phone": "555-0202"},
    {"name": "Dwight Schrute", "email": "dwight@dundermifflin.com", "dept": "Sales", "phone": "555-0303"},
]