Imagine you're building a log viewer for a production system. You have an API that returns logs in pages of 100 entries. You don't want to fetch all 50,000 logs into a single massive array—that's a great way to crash a browser tab or blow through your Node.js memory limit. You want to stream them: fetch a page, process it, and move to the next.

Here is a piece of code you might write if you're familiar with generators but haven't quite mastered the async variant yet:

async function* logStreamer() {
  let page = 1;
  while (true) {
    const response = await fetch(`https://api.logs.com/data?page=${page}`);
    const data = await response.json();
    if (data.logs.length === 0) break;
    yield data.logs;
    page++;
  }
}

const stream = logStreamer();
for (const batch of stream) {
  console.log("Processing batch:", batch);
}

When you run this, you'll notice something strange. Instead of seeing your logs, you'll see Processing batch: Promise { <pending> }. Then the loop ends abruptly, or it behaves unpredictably. It doesn't actually wait for the data to arrive.

The Promise-wrapped Iterator Problem

The issue here is a subtle but critical mismatch between the generator and the loop. An async function* doesn't return a standard Iterator; it returns an AsyncIterator. Every time the generator reaches a yield, it doesn't just give you a value; it returns a Promise that eventually resolves to that value.

A standard for...of loop is synchronous. It isn't designed to await the next value from an iterator. It just sees that the iterator gave it a Promise, treats that Promise as the value itself, and moves on immediately. Since the loop doesn't wait, it often finishes before the first network request even completes.

Implementing for await...of

To fix this, we need a loop that understands how to handle those Promises. JavaScript gives us a specific syntax for this: for await...of. This tells the engine, "Wait for the next Promise from this async generator to resolve before executing the loop body."

async function* logStreamer() {
  let page = 1;
  while (true) {
    const response = await fetch(`https://api.logs.com/data?page=${page}`);
    const data = await response.json();
    if (data.logs.length === 0) break;
    yield data.logs;
    page++;
  }
}

async function processLogs() {
  const stream = logStreamer();
  // Notice the 'await' keyword inside the loop declaration
  for await (const batch of stream) {
    console.log("Processing batch:", batch);
    // Now 'batch' is the actual array of logs, not a Promise
  }
}

processLogs();

Now the timing is correct. The loop pauses at the for await line, waits for logStreamer to fetch the page and yield it, and only then enters the loop body. I personally love this pattern because it keeps the "fetching logic" (the generator) entirely separate from the "processing logic" (the loop), all while keeping memory usage flat.

Handling Early Termination

One thing I've seen developers overlook is what happens if the consumer stops wanting data. If you break out of a for await...of loop, the async generator doesn't just vanish; it stays in a suspended state until it's garbage collected, which can occasionally lead to memory leaks or open connections in complex systems.

If your generator is doing something "heavy," like opening a database cursor or a file handle, you should use a try...finally block inside the generator. The finally block is guaranteed to run when the loop is broken or the generator is closed manually.

async function* databaseStreamer() {
  const cursor = await db.openCursor();
  try {
    while (await cursor.hasNext()) {
      yield await cursor.next();
    }
  } finally {
    await cursor.close();
    console.log("Cursor closed safely.");
  }
}

By combining async function* with for await...of, you've essentially built a pipeline. You can pull data from a source as fast as your consumer can handle it, without ever overloading your system's RAM.

📋 Practical Task

Build a Paginated User Search Streamer

You are tasked with creating a utility that searches for users across multiple pages of an API. Instead of returning one giant array, you must implement an async generator that streams the users.

Requirements:

• Create an async generator function called userSearchStreamer(query).
• The generator should simulate an API call using a helper function mockFetchUsers(query, page) (provided below).
• The generator should continue yielding pages of users until mockFetchUsers returns an empty array.
• Write an async function called runSearch() that uses a for await...of loop to consume the generator.
• Inside the loop, log the total number of users found across all batches.

// Provided helper: Do not modify this
async function mockFetchUsers(query, page) {
  const mockDatabase = [
    { name: 'Alice', query: 'js' }, { name: 'Bob', query: 'js' },
    { name: 'Charlie', query: 'js' }, { name: 'David', query: 'js' },
    { name: 'Eve', query: 'js' }, { name: 'Frank', query: 'js' },
  ];
  
  // Simulate pagination: 2 users per page
  const start = (page - 1) * 2;
  const end = start + 2;
  const results = mockDatabase.filter(u => u.query === query).slice(start, end);
  
  return new Promise(resolve => setTimeout(() => resolve(results), 100));
}

// Your code here:
async function* userSearchStreamer(query) {
  // TODO: Implement the generator
}

async function runSearch() {
  // TODO: Implement the for await...of loop and log total count
}

runSearch();