By now, you're probably comfortable with list comprehensions. They're elegant, they're fast, and they're a staple of "Pythonic" code. But there's a twin sibling to the list comprehension called the generator expression. They look almost identical—the only difference is that one uses square brackets [] and the other uses parentheses ()—but they behave fundamentally differently under the hood.

To see why this matters, let's build a small utility to process sensor readings. Imagine we have a stream of temperature data from a factory floor, and we only care about the readings that are over 100 degrees (the "overheat" events).

The instinctive approach (and why it's dangerous)

My first instinct is usually to just throw everything into a list. It's intuitive. I can see the data, I can slice it, and I can print it. Here is how I'd start writing this:

# Simulating a massive stream of 10 million readings
readings = range(10_000_000) 

# Find all readings over 100 (just as a dummy example)
overheats = [temp for temp in readings if temp > 100]

print(f"Found {len(overheats)} overheating events.")

Now, if this were a small script, I wouldn't even notice a problem. But in a production environment with truly massive datasets, I've actually crashed a server by doing this. Why? Because that list comprehension creates the entire list of overheating events in memory before it even lets me move to the next line of code. I'm asking Python to allocate enough RAM to hold millions of integers all at once.

Switching to lazy evaluation

This is where I make the pivot to a generator expression. I don't actually need the whole list of overheating events sitting in memory; I just need to be able to iterate through them one by one to process them. I'll change those square brackets to parentheses:

# Now using a generator expression
overheats_gen = (temp for temp in readings if temp > 100)

# I can still loop through it
for event in overheats_gen:
    # Process the event here
    if event > 1000: # Let's just stop early for the example
        break

The magic here is "lazy evaluation." The generator doesn't calculate the filtered list upfront. Instead, it returns a generator object that knows how to produce the next item only when you ask for it. Your memory usage stays flat, regardless of whether you're processing ten items or ten billion.

The "Wait, where's my data?" moment

I'll be honest: every time I start using generators, I make the same mistake. I try to treat the generator like a list. I'll do something like this:

overheats_gen = (temp for temp in readings if temp > 100)
print(overheats_gen[0]) # TypeError: 'generator' object is not subscriptable

I'll get a TypeError because generators don't support indexing. They don't "know" what the first or last element is until they've actually traveled through the sequence to find them. If you absolutely need to access elements by index or need to know the len() of the collection, you're stuck with a list comprehension (or you have to cast the generator to a list using list(overheats_gen), which brings you right back to the memory problem we just solved).

Choosing the right tool for the job

So, how do I decide which one to use? I follow a simple rule of thumb: if I'm going to iterate over the result once (like in a for loop or passing it into a function like sum() or max()), I use a generator expression. If I need to keep the data around to use it multiple times, or if I need to sort it or index it, I use a list comprehension.

📋 Practical Task

Memory-Efficient Log Filtering

You are tasked with analyzing a massive system log. Instead of loading all "ERROR" entries into a list—which would crash the system—you need to create a memory-efficient pipeline.

Write a script that does the following:

• Creates a list of strings called raw_logs containing a mix of "INFO", "WARNING", and "ERROR" messages (e.g., ["INFO: System boot", "ERROR: Disk full", "INFO: User login", "ERROR: Network down"]).
• Creates a generator expression that filters this list to only include the strings that start with "ERROR".
• Uses a for loop to iterate through that generator and print each error message in uppercase.

Ensure you use a generator expression () and not a list comprehension [] to ensure the filtering happens lazily.