How do I start a bunch of downloads without just doing them one by one?

If you use a standard foreach loop and await the download call inside it, you're not actually doing concurrent work. You're just doing sequential work with extra steps. To get true concurrency, you want to create a collection of Task objects first, and then tell C# to wait for all of them to finish.

I usually recommend Task.WhenAll for this. It's clean and it's efficient. Here is the pattern I use:

var downloadTasks = urls.Select(url => DownloadFileAsync(url));
await Task.WhenAll(downloadTasks);

Notice that I'm not await-ing DownloadFileAsync inside the Select. I'm just calling the method, which returns a Task (a promise of future work), and sticking that task into a list. The actual waiting happens only once at the end.

If I have 100 files to download, should I just fire off 100 tasks at once?

Honestly? No. Please don't do that. If you try to open 100 simultaneous TCP connections, you're likely to either get rate-limited by the server (who will think you're performing a DDoS attack) or you'll exhaust your own machine's socket pool. It's a classic rookie mistake that works fine with three files but crashes in production with three thousand.

The professional way to handle this is by using a SemaphoreSlim. Think of it as a nightclub bouncer; it only lets a specific number of tasks "into the club" at one time. Here's how I'd implement a throttle:

private readonly SemaphoreSlim _bouncer = new SemaphoreSlim(4); // Only 4 at a time

async Task DownloadFileAsync(string url)
{
    await _bouncer.WaitAsync(); 
    try 
    {
        // Actual HttpClient logic here
        var data = await _httpClient.GetByteArrayAsync(url);
        await File.WriteAllBytesAsync(GetFileName(url), data);
    }
    finally 
    {
        _bouncer.Release(); // Let the next task in
    }
}

What happens if one download fails? Does the whole process crash?

This is where Task.WhenAll can be a bit tricky. If one of the tasks throws an exception, Task.WhenAll will eventually throw that exception. But here's the catch: the other downloads are still running in the background. They don't just stop because one failed.

If you want your downloader to be resilient, you shouldn't let the exception bubble up to the WhenAll call. Instead, wrap the logic inside the task in a try-catch block. I usually return a small result object so I can report exactly which files failed at the end without killing the entire process.

public record DownloadResult(string Url, bool Success, string Error = "");

async Task<DownloadResult> DownloadFileAsync(string url)
{
    try 
    {
        await _httpClient.GetAsync(url);
        return new DownloadResult(url, true);
    }
    catch (Exception ex)
    {
        return new DownloadResult(url, false, ex.Message);
    }
}

By doing this, Task.WhenAll returns an array of DownloadResult objects. You can then simply filter that list to see who failed and maybe retry just those specific ones.

📋 Practical Task

Exercise: The High-Res Wallpapers Batch Downloader

Your goal is to build a console application that downloads a list of high-resolution images from a set of URLs. To make this a real-world scenario, you must implement the following constraints:

• The Source: Create a List<string> containing 10-15 URLs of images (you can use placeholder image services like picsum.photos).
• The Throttle: Use a SemaphoreSlim to ensure that no more than 3 downloads are happening simultaneously.
• The Resilience: Wrap each download in a try-catch block. If a download fails, the program should not crash; instead, it should log the failed URL to the console and continue with the remaining files.
• The Completion: Use Task.WhenAll to wait for all downloads to conclude. Once finished, print a summary showing the total number of successful downloads and the total number of failures.

Bonus: Implement a simple timeout using a CancellationTokenSource so that if a single image takes longer than 10 seconds to download, that specific task is cancelled.