I want to show you a piece of code that looks perfectly fine at first glance, but will absolutely tank your production server if you run it against a real-world dataset. I've seen this exact mistake made by junior devs (and even some seniors) more times than I can count.

def get_critical_logs(filename):
    critical_errors = []
    with open(filename, 'r') as f:
        for line in f:
            if "CRITICAL" in line:
                critical_errors.append(line)
    return critical_errors

# Imagine 'huge_system.log' is 15GB
logs = get_critical_logs("huge_system.log") 
for log in logs:
    print(log)

The Memory Wall

The logic here is sound, right? We open a file, filter for the word "CRITICAL", and return a list of those lines. But here is the problem: the critical_errors list lives in your RAM. If your log file is 15GB and 2GB of that consists of critical errors, Python will try to allocate 2GB of memory to hold that list before it ever reaches the for log in logs loop.

If your server only has 1GB of free RAM, the OS is going to kill your process with a MemoryError. You're trying to swallow the entire ocean before you've even taken a single sip.

Swapping Return for Yield

This is where yield comes in. When you replace return with yield, you aren't writing a standard function anymore—you're creating a generator function. A generator doesn't compute the entire list and hand it over; instead, it pauses its execution and hands back one value at a time, only when requested.

def get_critical_logs(filename):
    with open(filename, 'r') as f:
        for line in f:
            if "CRITICAL" in line:
                yield line # This is the magic part

# Now, 'logs' isn't a list; it's a generator object
logs = get_critical_logs("huge_system.log") 
for log in logs:
    print(log)

Notice that I removed the critical_errors list entirely. Now, the function finds one critical line, yields it to the loop, and then just... sits there. It stays paused exactly where it left off. When the loop asks for the next item, the function wakes up, finds the next match, and yields that. Your memory usage stays flat, regardless of whether the file is 10MB or 10TB.

The Trade-off of Lazy Evaluation

We call this "lazy evaluation" because Python is being lazy—it refuses to do any work until you absolutely force it to by iterating over it. This is incredibly powerful, but there is one big catch you need to remember: generators are one-time-use.

If you try to loop through logs a second time, it will be empty. The "cursor" has already reached the end of the file and the generator is exhausted. If you need the data multiple times, you'll either need to call the generator function again or actually convert it to a list using list(logs)—though doing the latter brings us right back to the memory problem we just solved.

• return: "Here is the entire finished product. I'm done now."
• yield: "Here is one piece. I'll wait right here until you need the next one."

📋 Practical Task

Building a Large-Scale CSV Row Filter

You are tasked with processing a massive CSV file containing user transactions. The file is too large to fit in memory, and you only care about transactions that are over $1,000.00.

Your Goal: Create a generator function called filter_high_value_transactions(data_source). Instead of a real file, the data_source will be a list of strings (simulating lines in a file) to make the exercise easier to run.

Requirements:

• The function must yield only the lines that contain a value greater than 1000.
• The lines are formatted as "UserID,Amount" (e.g., "user_1,150.00").
• Do not use a list to collect the results inside the function.

# Test data
mock_csv_data = [
    "user_1,150.00",
    "user_2,1200.50",
    "user_3,45.00",
    "user_4,3000.00",
    "user_5,999.99"
]

# Your generator function here
def filter_high_value_transactions(data_source):
    # TODO: Implement the generator
    pass

# Testing the generator
high_values = filter_high_value_transactions(mock_csv_data)
for transaction in high_values:
    print(f"High value found: {transaction}")