Why can't I just use a for-loop with the requests library?

You technically can, but you're wasting an enormous amount of time. When you use requests.get(), your code stops everything and just sits there—literally doing nothing—while it waits for the server to send a response back over the wire. If you have 100 URLs and each takes one second, you've just spent a minute and a half staring at a blinking cursor.

By using asyncio and aiohttp, we change the game. Instead of waiting for one request to finish before starting the next, we fire off a request and immediately tell Python, "Hey, while we're waiting for that server to wake up, go ahead and start the next five requests." It's the difference between a waiter who takes one order, walks it to the kitchen, and waits for the food to be cooked before greeting the next customer, versus a waiter who takes five orders in a row and then delivers the plates as they come out of the kitchen.

How do I stop myself from accidentally DDOSing the site I'm scraping?

This is where a lot of beginners trip up. If you have 1,000 URLs and you use asyncio.gather(*tasks) without any constraints, Python will attempt to open 1,000 simultaneous TCP connections. Most servers will see that as an attack and either block your IP immediately or just crash. I've definitely seen a few internal company tools get blacklisted because someone forgot this part.

The solution is a Semaphore. Think of it as a bouncer at a club who only lets a specific number of people in at a time. Here is how I usually implement it:

import asyncio
import aiohttp

async def fetch_url(session, url, semaphore):
    async with semaphore: # This ensures only X requests run at once
        async with session.get(url) as response:
            return await response.text()

async def main():
    urls = ["https://example.com/page1", "https://example.com/page2", "..."]
    semaphore = asyncio.Semaphore(10) # Limit to 10 concurrent requests
    
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_url(session, url, semaphore) for url in urls]
        results = await asyncio.gather(*tasks)
        return results

What's the best way to handle a few failed requests without crashing the whole script?

In a synchronous loop, a try/except block is straightforward. In an async environment, if one of your gathered tasks raises an unhandled exception, it can get messy. You don't want one 404 error or a timeout on the 50th URL to kill the remaining 950 requests you've already started.

I recommend wrapping the logic inside your worker function. Instead of letting the exception bubble up to asyncio.gather, catch it right where it happens and return a specific value (like None or an error message) so you can filter it out later. Check out this pattern:

async def fetch_url(session, url, semaphore):
    async with semaphore:
        try:
            async with session.get(url, timeout=10) as response:
                # Raise an exception for 4xx or 5xx errors
                response.raise_for_status() 
                return await response.text()
        except Exception as e:
            print(f"Failed to fetch {url}: {e}")
            return None # Return None so we know this one failed

Then, when you get your list of results back from gather(), you can just do a quick list comprehension to remove the None values: valid_results = [r for r in results if r is not None].

📋 Practical Task

Exercise: Building a Concurrent Product Price Tracker

Your goal is to build a scraper that fetches the HTML content of 20 different product pages from a mock e-commerce site (you can use https://books.toscrape.com/catalogue/page-X.html where X is 1 through 20).

• Use aiohttp for the requests and asyncio for concurrency.
• Implement an asyncio.Semaphore to ensure no more than 5 requests are happening at any single moment.
• Wrap your request logic in a try/except block to handle potential connection errors or timeouts gracefully.
• After gathering all responses, print the total number of successfully retrieved pages and the average size (in bytes) of the HTML content returned.