Let's say we're building a tool to track prices for a few hundred different products across a competitor's site. You've got your list of URLs, and you know exactly how to parse the HTML to find the price tag. On the surface, this seems like a trivial task. But as you move from scraping five pages to five hundred, you'll quickly realize that how you handle the execution flow is the difference between a professional tool and a script that gets your IP banned in ten seconds.

The Sequential Slog

The first instinct for most is to just wrap the request in a simple loop. You've already learned about async and await, so you probably write something like this:

foreach (var url in productUrls)
{
    var html = await httpClient.GetStringAsync(url);
    var price = ParsePrice(html);
    results.Add(price);
}

This is "safe," but it's painfully slow. You're spending 99% of your execution time idling, waiting for a server halfway across the world to send back a packet of data before you even think about requesting the next page. If each request takes 500ms, 1,000 pages will take over eight minutes. In the world of data collection, that's an eternity. You're underutilizing your machine and your bandwidth.

The "Fire-and-Forget" Disaster

To fix the speed, you might be tempted to just kick off all the tasks at once. I've seen a lot of developers do this when they first discover Task.WhenAll. It feels like magic because the speed increase is immediate:

var tasks = productUrls.Select(url => httpClient.GetStringAsync(url));
var allHtml = await Task.WhenAll(tasks);

Here is where things break. By doing this, you aren't just "being fast"—you're launching a Distributed Denial of Service (DDoS) attack on the target server. If you have 1,000 URLs, you're attempting to open 1,000 concurrent TCP connections. The remote server will likely see this spike, decide you're a malicious bot, and return 429 Too Many Requests or just drop your connection entirely. Even if the server handles it, your own machine might struggle with socket exhaustion. It's an "all-or-nothing" approach that lacks any semblance of empathy for the infrastructure you're hitting.

Controlling the Chaos with Parallel.ForEachAsync

The professional way to handle this is through bounded concurrency. We want to be fast, but we want to stay under the radar and keep our resource usage predictable. In modern C#, the cleanest way to achieve this is Parallel.ForEachAsync.

Instead of choosing between "one at a time" and "all at once," we define a MaxDegreeOfParallelism. This lets us tell the runtime: "I want to process this list as fast as possible, but never have more than, say, 5 requests in flight at any given moment."

var options = new ParallelOptions 
{ 
    MaxDegreeOfParallelism = 5 
};

await Parallel.ForEachAsync(productUrls, options, async (url, token) => 
{
    try 
    {
        var html = await httpClient.GetStringAsync(url, token);
        var price = ParsePrice(html);
        
        // Be careful here: List<T> is not thread-safe!
        lock(results) 
        {
            results.Add(price);
        }
    }
    catch (HttpRequestException ex)
    {
        Console.WriteLine($"Failed to fetch {url}: {ex.Message}");
    }
});

Notice the lock around the results list. This is a critical detail. While Parallel.ForEachAsync handles the threading for you, the code inside the lambda is running on multiple threads. If two threads try to Add to a standard List at the exact same millisecond, you'll end up with corrupted data or a NullReferenceException. If you hate locks, you could use a ConcurrentBag<T>, but a simple lock is often more than enough for this kind of work.

By using this pattern, you've found the sweet spot. You're utilizing your network effectively, you're not crashing the target server, and your application remains stable. It's about balancing aggression with reliability.

📋 Practical Task

Exercise: The GadgetStore Price Tracker Throttle

You are tasked with writing a scraper for "GadgetStore". You have a list of 50 product URLs. You must implement a solution that uses Parallel.ForEachAsync to fetch the content of these pages.

Requirements:

• Create a List<string> containing 50 dummy URLs (e.g., "https://gadgetstore.com/p/1" through "https://gadgetstore.com/p/50").
• Use a single HttpClient instance.
• Set the MaxDegreeOfParallelism to exactly 4.
• Since you don't have a real server, wrap your GetStringAsync call in a try-catch block and, instead of actual parsing, simply add the URL to a ConcurrentBag<string> or a locked List<string> to signify a "successful" scrape.
• Print the total number of successfully "scraped" URLs to the console at the end.