A few years back, I was maintaining a legacy system that exported massive daily dumps of user activity logs into JSON files. Everything was fine until our data grew to a point where the network transfer between the app server and the storage bucket started timing out. I spent a frustrating morning trying to optimize the network config, only to realize we were essentially sending the same repetitive keys—"timestamp", "event_type", "user_id"—millions of times per file. I spent ten minutes implementing a compression wrapper, and suddenly those 500MB files shrunk to 40MB. The timeouts vanished, and our storage costs plummeted.

In Python, when you need to shrink data, you'll typically reach for either zlib or gzip. While they are closely related—gzip actually uses the zlib library under the hood—they serve different purposes. Think of zlib as the tool for raw data streams in memory, and gzip as the tool for creating actual files that you can open with tools like WinZip or gunzip on Linux.

Working with In-Memory Buffers via zlib

If you're sending data over a socket or storing a blob in a database, you don't want the overhead of a file header. You just want the bytes to be smaller. That's where zlib shines. It takes a bytes object and returns a compressed bytes object.

import zlib

# Imagine this is a repetitive log entry or a large JSON string
data = b"USER_LOGIN: admin | IP: 192.168.1.1\n" * 1000
print(f"Original size: {len(data)} bytes")

# Compress the data
compressed = zlib.compress(data, level=6) 
print(f"Compressed size: {len(compressed)} bytes")

# Bring it back to life
decompressed = zlib.decompress(compressed)
assert decompressed == data

You'll notice the level argument. It ranges from 0 (no compression) to 9 (maximum compression). I usually stick with 6; it's the default because it provides a sweet spot between CPU usage and the resulting file size. If you go to 9, you might save a few more kilobytes, but you'll spend significantly more CPU cycles getting there, which is rarely worth it in a production environment.

Managing Compressed Files with gzip

When you need to save a file to disk that other programs can recognize as a .gz file, zlib isn't enough because it lacks the necessary file headers. This is why we use the gzip module. The beauty of gzip is that it mimics Python's built-in open() function almost perfectly.

import gzip

content = "This is a large amount of text that we want to store efficiently.".encode('utf-8')

# Writing a compressed file
with gzip.open('system_logs.txt.gz', 'wb') as f:
    f.write(content)

# Reading it back
with gzip.open('system_logs.txt.gz', 'rb') as f:
    file_content = f.read()
    print(file_content.decode('utf-8'))

One thing to keep in mind: gzip.open defaults to binary mode ('rb' or 'wb'). If you want to handle text directly, you can pass mode='wt' (write text) or mode='rt' (read text), and Python will handle the encoding/decoding for you. I personally prefer binary mode for compression tasks because it makes it explicit that you're dealing with a byte stream, which helps avoid those annoying TypeError bugs when you're mixing strings and bytes.

📋 Practical Task

Exercise: Building a Compressed Log Archiver

You are tasked with creating a utility that helps a server clean up its logs. Write a script that does the following:

• Create a dummy log file named server.log containing 10,000 lines of the text: "INFO: System heartbeat detected at 2023-10-27 10:00:00\n".
• Read that file and use the gzip module to create a compressed version called server.log.gz.
• Calculate the size of both the original file and the compressed file using os.path.getsize().
• Print a message stating the original size, the compressed size, and the percentage of space saved.
• Finally, verify the integrity of the compressed file by reading it back and printing the first line to ensure the data is intact.