Skip to Content
Course content

218: Building a Multi-Threaded File Downloader

Click on the "Edit" button in the top corner of the screen to edit your slide content.

I've seen this mistake a dozen times in code reviews: a developer decides to speed up a file downloader by wrapping the download logic in a FixedThreadPool with 20 threads, thinking that because they have 20 threads, the file will download 20 times faster. They write a loop that triggers 20 separate download requests for the same file, and then they wonder why their hard drive is filled with 20 identical copies of the same 500MB zip archive, or why the server has banned their IP for spamming requests.

The Myth: More Threads Equals a Faster Pipe

Here is the concrete reality: if you are downloading a single file from a single server, spawning multiple threads to "download the file" doesn't magically widen your internet bandwidth. If your connection is 100Mbps, that's your ceiling. In fact, if you simply launch multiple threads to download the entire file, you're just wasting resources and potentially triggering a DDoS protection alarm on the server side. You aren't parallelizing the work; you're just duplicating it.

The Reality: Leveraging HTTP Range Requests

To actually gain a performance boost—specifically when dealing with large files and servers that support it—you don't download the file multiple times. Instead, you download different chunks of the same file simultaneously. This is how "Download Manager" software actually works.

The secret is the Range header in the HTTP protocol. By sending a header like Range: bytes=0-499, you tell the server, "Don't give me the whole thing; just give me the first 500 bytes." If you do this across four threads for a 2MB file, each thread handles 500KB. I'll admit, for small files, the overhead of managing the threads actually makes this slower. But for a 2GB ISO image? It's a game changer.

Coordinating Chunks with RandomAccessFile

The tricky part isn't the downloading—it's the writing. If you use a standard FileOutputStream, you're stuck writing sequentially from the start of the file. If Thread B finishes its chunk before Thread A, you can't just write Thread B's data, or you'll overwrite the beginning of the file with the middle of the content.

This is where RandomAccessFile comes in. It allows you to "seek" to a specific byte offset in the file. I like to think of it as a record player needle; you can jump directly to the 1GB mark and start writing there, regardless of whether the first 999MB have arrived yet.

// A simplified snippet of how a single chunk task looks
public class DownloadChunk implements Runnable {
    private final String url;
    private final long startByte;
    private final long endByte;
    private final RandomAccessFile file;

    public DownloadChunk(String url, long start, long end, RandomAccessFile file) {
        this.url = url;
        this.startByte = start;
        this.endByte = end;
        this.file = file;
    }

    @Override
    public void run() {
        try {
            HttpClient client = HttpClient.newHttpClient();
            HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create(url))
                .header("Range", "bytes=" + startByte + "-" + endByte)
                .build();

            HttpResponse<InputStream> response = client.send(request, HttpResponse.BodyHandlers.ofInputStream());
            
            // Use a synchronized block or a lock because multiple threads 
            // are sharing the same RandomAccessFile object
            synchronized (file) {
                file.seek(startByte);
                response.body().transferTo(Channels.newOutputStream(file.getFD()));
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Note that I used synchronized(file). Even though seek() and write() happen at different offsets, the file pointer is shared. If Thread A seeks to 0 and then Thread B seeks to 1000 before Thread A actually writes, Thread A will accidentally write its data at position 1000. Always lock your shared file resource.




📋 Practical Task

Build a Parallel Image Recovery Tool

Your task is to create a program that downloads a large image file (or any large file from a public URL) using exactly four threads. You cannot simply download the file four times; you must split the file into four equal byte-ranges.

  • First, perform a "HEAD" request to the URL to determine the total Content-Length of the file.
  • Calculate the start and end byte offsets for four equal segments.
  • Use an ExecutorService to manage four DownloadChunk tasks.
  • Use a RandomAccessFile to assemble the pieces into a single file on your local disk.
  • Verify the final file size matches the Content-Length reported by the server.

Hint: If you're testing with a random website, make sure the server supports Range requests. Most modern CDNs and S3 buckets do, but some basic web servers might ignore the header and send the whole file anyway.

Rating
0 0

There are no comments for now.

to be the first to leave a comment.