A few years ago, I was reviewing a PR for a colleague who was building a dashboard to export thousands of user records from a legacy API. They had written a loop that fetched every single page of data, pushed them all into one massive array using Promise.all(), and then processed that array. It worked fine in staging with ten users, but the second it hit production with 50,000 records, the Node process hit a heap out-of-memory error and crashed. The mistake wasn't the logic; it was trying to hold the entire "future" of the data in memory at once instead of treating the data as a stream.

This is exactly where async iterators come in. Instead of waiting for every single promise to resolve before you start working, async iterators allow you to process each chunk of data the moment it arrives, while the rest of the data is still being fetched. It's the difference between waiting for a whole movie to download before watching it and streaming it via Netflix.

Building an Async Generator

To create an async iterable, the most ergonomic way is using an async generator. You'll recognize the syntax—it's an async function, but with an asterisk: async function*. Inside, instead of returning a single value, you yield values.

Let's look at how we could have fixed that pagination disaster. Instead of an array, we create a generator that yields one page of results at a time:

async function* fetchUserPages(apiUrl) {
  let nextPage = 1;
  let hasMore = true;

  while (hasMore) {
    const response = await fetch(`${apiUrl}?page=${nextPage}`);
    const data = await response.json();
    
    yield data.users; // We "pause" here and hand the data to the loop

    if (data.currentPage < data.totalPages) {
      nextPage++;
    } else {
      hasMore = false;
    }
  }
}

The magic here is that yield doesn't just return a value; it returns a promise that resolves to that value, and it pauses the execution of the generator. The generator stays "alive," remembering exactly where it left off until the consumer asks for the next item.

Consuming Streams with for-await-of

Now, you can't use a standard for...of loop here. If you try, JavaScript will complain that the object is not iterable because it's an async iterable. This is why we have the for await...of statement.

This loop tells JavaScript: "I know the next item in this collection is a promise. Please wait for it to resolve before executing the body of the loop."

async function processAllUsers() {
  const userStream = fetchUserPages('https://api.example.com/users');

  try {
    for await (const users of userStream) {
      // This block only runs once a page has been successfully fetched
      console.log(`Processing a batch of ${users.length} users...`);
      await saveToDatabase(users); 
    }
    console.log('All users processed successfully.');
  } catch (err) {
    console.error('The stream broke:', err);
  }
}

I love this pattern because it keeps your memory footprint tiny. Whether you are processing ten users or ten million, your application only ever holds one page of data in memory at a time. Plus, if saveToDatabase is also async, the loop naturally throttles itself. It won't fetch page 2 until you've finished processing page 1, preventing you from accidentally DDoS-ing your own API.

One quick word of caution: for await...of only works inside async functions (or top-level await modules). Also, keep in mind that since it awaits each item sequentially, it's not the right tool if you need to fire off ten requests simultaneously for speed—but for sequential streams, it's unbeatable.

📋 Practical Task

Exercise: Building a Simulated Log File Streamer

Imagine you are building a CLI tool that reads a massive system log file. Since the file is too large to load into memory, you need to stream it line-by-line.

Your Task:

• Create an async generator function called logStreamer that takes an array of strings (representing the lines of a file).
• Inside the generator, use a for...of loop to iterate through the strings.
• For each string, simulate a network/disk delay by awaiting a Promise that resolves after 100ms.
• yield the line after the delay.
• Write an async function called analyzeLogs that uses a for await...of loop to consume the logStreamer.
• Inside analyzeLogs, check if the line contains the word "ERROR". If it does, console.log it with a prefix: "Alert: [the line]".

Test your code with this dataset:

const mockLogs = [
  "INFO: System started",
  "INFO: User logged in",
  "ERROR: Database connection failed",
  "INFO: Retrying connection",
  "ERROR: Timeout reached",
  "INFO: System shutdown"
];