Up until now, we've been playing around with Python's internal logic. But the real power of the language often comes from how it interacts with the rest of the web. Today, we're going to build a simple web scraper. I've found that the best way to learn this is by actually trying to pull data from a page and failing a few times until the selectors click.

We're going to target books.toscrape.com. It's a sandbox site specifically designed for practicing scraping, so we don't have to worry about getting our IP banned for hitting a production server too hard.

Setting up the requests and the parser

To do this, we need two tools: requests to fetch the page and BeautifulSoup to make sense of the HTML. I always start by importing these and making a basic GET request. I like to print the status code immediately—there's nothing worse than spending an hour debugging a parser only to realize the server was giving you a 403 Forbidden error the whole time.

import requests
from bs4 import BeautifulSoup

url = "http://books.toscrape.com/"
response = requests.get(url)

if response.status_code == 200:
    print("Successfully connected!")
    soup = BeautifulSoup(response.text, 'html.parser')
else:
    print(f"Failed to retrieve page. Status: {response.status_code}")

Hunting for the book data

Now we need to figure out where the data lives. If you right-click a book title on that site and "Inspect" it, you'll see that each book is wrapped in an <article class="product_pod">. My strategy here is always to find the "container" first, and then loop through those containers to find the specific details inside.

# Find all the book containers
books = soup.find_all('article', class_='product_pod')

for book in books:
    # The title is inside an 'a' tag within an 'h3'
    title = book.h3.a['title']
    
    # The price is in a 'p' tag with a specific class
    price = book.find('p', class_='price_color').text
    
    print(f"Title: {title} | Price: {price}")

Dealing with a common "NoneType" blunder

Here is where I usually trip up. Let's say I wanted to scrape the "Availability" status. I might try something like this:

# My mistake: Assuming the tag is always there or correctly named
stock = book.find('p', class_='instock').text
print(f"Stock: {stock}")

If I run this and the site suddenly changes a class name or a specific item is missing that tag, Python will throw an AttributeError: 'NoneType' object has no attribute 'text'. This happens because .find() returns None if it doesn't find a match, and you can't call .text on None. It's a classic "scraping crash."

To fix this, I always use a conditional check or a helper variable. It makes the code slightly longer, but it keeps the scraper from dying halfway through a list of a thousand items.

stock_tag = book.find('p', class_='instock availability')
stock_status = stock_tag.text.strip() if stock_tag else "Unknown"
print(f"Stock: {stock_status}")

Organizing the results into a list

Printing to the console is fine for a quick test, but in a real project, you'll want this data in a structure you can actually use, like a list of dictionaries. This makes it easy to export to a CSV or a database later on.

all_books = []

for book in books:
    title = book.h3.a['title']
    price = book.find('p', class_='price_color').text
    stock_tag = book.find('p', class_='instock availability')
    
    all_books.append({
        'title': title,
        'price': price,
        'stock': stock_tag.text.strip() if stock_tag else "Unknown"
    })

# Now we have a clean list of data
print(all_books[0]) 
# Output: {'title': 'A Light in the Attic', 'price': '£51.77', 'stock': 'In stock'}

📋 Practical Task

Exercise: Build a Quote Collector from Quotes to Scrape

Using the concepts from this lesson, create a scraper for http://quotes.toscrape.com. Your script should perform the following tasks:

• Fetch the main page of the site.
• Find all the quote containers (look for the <div class="quote"> tags).
• Extract the text of the quote and the author's name.
• Store these pairs in a list of dictionaries (e.g., [{'quote': '...', 'author': '...'}, ...]).
• Print the total number of quotes found and the first dictionary in your list to verify it works.