I remember the first time I crashed a production server because of something as simple as a log file. I had a service that logged every single incoming request. It worked great for a week, then suddenly, the server stopped accepting connections. The disk was 100% full. I had one massive 40GB text file that had simply eaten everything. It was a humbling moment, and it's why we're talking about rotation today.

The Growing Giant

Let's simulate a similar disaster. I've got a script here that mimics a high-frequency sensor reading. It's just dumping data into a file as fast as it can. I'll start with a standard setup using logging.basicConfig.

import logging
import time

logging.basicConfig(
    filename='sensor_data.log', 
    level=logging.INFO, 
    format='%(asctime)s - %(message)s'
)

# Simulate a sensor that logs way too much
while True:
    logging.info("Sensor Reading: Temp=22.4C, Humidity=45%, Pressure=1013hPa")
    time.sleep(0.01) 

If I run this for a few minutes, sensor_data.log starts ballooning. In a real environment, this is a ticking time bomb. I can't just manually delete the file while the program is running—Python holds a handle on it, and on some OSs, that's a nightmare to clean up. I need the system to manage its own footprint.

Switching to a Managed Handler

Since basicConfig is too blunt for this, I need to get my hands on the logging.handlers module. Specifically, the RotatingFileHandler. The idea is that once the file hits a certain size, Python will close it, rename it (like sensor_data.log.1), and start a fresh sensor_data.log.

Let's rewrite the setup. I'll set a very small maxBytes limit just so we can actually see the rotation happening without waiting for gigabytes to accumulate.

import logging
from logging.handlers import RotatingFileHandler
import time

# I'll create a logger object instead of using the root logger via basicConfig
logger = logging.getLogger("SensorLogger")
logger.setLevel(logging.INFO)

# maxBytes=2000 means it rotates after about 2KB
# backupCount=5 means it keeps 5 old files before deleting the oldest one
handler = RotatingFileHandler("sensor_data.log", maxBytes=2000, backupCount=5)
logger.addHandler(handler)

while True:
    logger.info("Sensor Reading: Temp=22.4C, Humidity=45%, Pressure=1013hPa")
    time.sleep(0.01)

Observing the Shuffle

Now, if I run this and keep my file explorer open, I'll see something interesting. The sensor_data.log grows, then suddenly it vanishes and sensor_data.log.1 appears. Then sensor_data.log starts growing again. When it hits the limit again, .1 becomes .2, and the current log becomes .1.

Wait, I noticed something. If I set backupCount=0, the rotation still happens, but it just overwrites the current file without keeping any backups. That's basically useless if you actually need to debug something from an hour ago. The backupCount is what gives us our safety net.

What about time-based rotation?

Depending on the project, size isn't always the right metric. Some of my colleagues prefer "daily" logs. If I want a new file every midnight regardless of size, RotatingFileHandler won't cut it. For that, I'd swap it for TimedRotatingFileHandler.

The logic is almost identical, but instead of maxBytes, you provide when='midnight' or when='D' (for days). It's a design choice: do you care more about disk space (size-based) or chronological auditing (time-based)? In most high-traffic systems, I actually use a combination or stick to size-based rotation to prevent the "disk full" catastrophe I mentioned earlier.

📋 Practical Task

Build a High-Frequency Log Governor

You are tasked with creating a logging system for a simulated network packet sniffer. The sniffer generates a massive amount of data, and you must ensure it never consumes more than a fixed amount of disk space.

• Create a logger named "PacketSniffer".
• Implement a RotatingFileHandler that writes to network.log.
• Configure the handler so that it rotates when the file reaches 10 KB (10,240 bytes).
• Ensure that the system keeps exactly 3 backup files.
• Write a loop that logs the string "Packet captured: Source=192.168.1.1, Dest=8.8.8.8, Size=64 bytes" repeatedly until at least 4 rotation files (the current log + 3 backups) are visible in your directory.