A few years ago, I worked with a developer who was tasked with monitoring a competitor's pricing for about fifty different hardware components. For two weeks, he spent the first hour of every morning manually refreshing pages, copying prices into an Excel sheet, and sighing deeply. He was a great engineer, but he was treating himself like a data-entry clerk. It wasn't until I showed him how to use Go to automate the process that he actually started enjoying the project. He went from spending sixty minutes on manual labor to running a single command that took three seconds. That's the power of a well-built scraper.

At its core, web scraping is just two things: making an HTTP request to get the HTML of a page, and then parsing that HTML to find the specific bits of data you actually care about. While Go's standard library is fantastic, parsing raw HTML with golang.org/x/net/html can feel like trying to eat soup with a fork—it's possible, but it's tedious. Instead, we'll use goquery, which brings a jQuery-like syntax to Go, making it significantly easier to select elements using CSS selectors.

Fetching the Page and Managing Resources

Before we can extract data, we need the HTML. We use the net/http package for this. The most critical thing to remember here is managing the response body. If you don't close the response body, you'll leak network connections, and your program will eventually crash or hang when you try to scrape hundreds of pages.

package main

import (
	"log"
	"net/http"
	"github.com/PuerkitoBio/goquery"
)

func main() {
	// We make a simple GET request
	res, err := http.Get("https://example-blog.com/articles")
	if err != nil {
		log.Fatal(err)
	}
	// Always defer the close immediately after checking for errors
	defer res.Body.Close()

	if res.StatusCode != 200 {
		log.Fatalf("status code error: %d %s", res.StatusCode, res.Status)
	}

	// Load the HTML document
	doc, err := goquery.NewDocumentFromReader(res.Body)
	if err != nil {
		log.Fatal(err)
	}
}

Targeting Data with CSS Selectors

Once we have the doc object, we can treat the webpage like a searchable tree. If you've ever used Chrome DevTools to "Inspect" an element, you already know how this works. You find the class or ID of the element you want, and you tell Go to find all instances of it. I usually prefer targeting specific classes because IDs are often dynamically generated by modern frameworks and can change on every page load.

Let's say we're scraping a blog where every article title is wrapped in an <h2> tag with a class of .post-title. We can iterate through these using the Find and Each methods:

doc.Find(".post-title").Each(func(i int, s *goquery.Selection) {
	title := s.Text()
	link, exists := s.Find("a").Attr("href")
	if exists {
		log.Printf("Article %d: %s - %s\n", i, title, link)
	}
})

Notice how we use s.Find("a").Attr("href") inside the loop. This is a nested search; we are looking for the anchor tag inside the specific title element we've already found. It's a clean, hierarchical way to traverse the DOM.

Handling the Ethics of Scraping

I'll be honest: it's very easy to accidentally DDoS a small website if you put your scraping logic inside a tight for loop with goroutines. Most servers will see a thousand requests per second from a single IP and immediately block you. To avoid this, you should always implement a small delay between requests using time.Sleep.

Furthermore, always check the robots.txt file of the domain (e.g., example.com/robots.txt). It tells you which parts of the site the owner doesn't want bots to touch. Respecting these rules isn't just about being "polite"—it's about ensuring your scraper doesn't get your IP blacklisted by a firewall before you've even finished your first data set.

📋 Practical Task

Build a Book Title and Price Extractor

Your task is to build a scraper that targets the sandbox site books.toscrape.com. This site is specifically designed for developers to practice scraping without getting banned.

• Create a program that fetches the home page of books.toscrape.com.
• Use goquery to locate all product containers (hint: look for the <article class="product_pod"> elements).
• For each product, extract the Title (found within the <h3> <a> tag) and the Price (found within the <p class="price_color"> tag).
• Print the results to the console in the format: "Book: [Title] | Price: [Price]".
• Ensure you properly handle the HTTP response body closure to avoid memory leaks.