I've been thinking about how we handle large datasets in Scala. Up until now, we've mostly worked with List or Vector. They're great, but they have a fatal flaw: they live in memory. If you try to load a 10GB log file into a List[String], your JVM is going to throw an OutOfMemoryError faster than you can type Ctrl+C.

I want to build a simple log analyzer that finds all "ERROR" entries in a file and counts them. Let's start with the "naive" way and see where it breaks.

The Memory Wall

import zio._

object LogAnalyzer extends ZIOAppDefault {
  def run = {
    for {
      // Imagine this file is massive
      lines <- ZIO.attempt(scala.io.Source.fromFile("app.log").getLines().toList)
      errors = lines.filter(_.contains("ERROR"))
      _ <- Console.printLine(s"Found ${errors.size} errors")
    } yield ()
  }
}

The problem here is .toList. It forces the entire file into RAM. If I'm running this on a production server with a strict heap limit, this code is a ticking time bomb. I need a way to process the file line-by-line, but I still want the power of ZIO—like timeouts, retries, and concurrency. This is exactly why ZStream exists.

Switching to a Flow Mindset

Instead of thinking of the data as a collection, I need to think of it as a pipeline. In ZIO Streams, we don't "get" the data; we describe how the data should flow. Let's try to rewrite this using ZStream.

import zio._
import zio.stream._

object LogAnalyzer extends ZIOAppDefault {
  def run = {
    val pipeline = ZStream.fromFileName("app.log")
      .via(ZPipeline.utf8Decode)
      .filter(_.contains("ERROR"))
      .map(_ => 1)
      .runFold(0)(_ + _)

    for {
      count <- pipeline
      _     <- Console.printLine(s"Found $count errors")
    } yield ()
  }
}

Notice what happened here. I didn't call a method that returns a List. ZStream.fromFileName creates a description of a stream. The filter and map calls are also just descriptions. Nothing actually happens until I hit runFold. This is "lazy" evaluation on steroids.

Wait, what is a Chunk?

While playing with this, I noticed that ZIO Streams don't actually move elements one-by-one. If they did, the overhead of the effect system would kill performance. Instead, they use Chunk.

I'll add a tap to the stream to see what's happening under the hood:

ZStream.fromIterable(1 to 100)
  .chunked(10) // Group them into chunks of 10
  .tap(chunk => Console.printLine(s"Processing chunk of size: ${chunk.size}"))
  .runDrain

If you run this, you'll see it processes 10 elements at a time. This is the secret sauce. ZIO streams balance the memory safety of "one-at-a-time" processing with the performance of "batch" processing. If you're sending data over a network, you don't want to send one byte per packet; you want a chunk.

Handling the "Infinite" Problem

One thing that tripped me up early on was trying to use runCollect on a stream that might be infinite (like a socket connection). runCollect tries to put everything into a Chunk (which eventually becomes a List), bringing me right back to the OutOfMemoryError I started with.

If you are dealing with a stream that doesn't have a clear end, you can't "collect" it. You have to "consume" it. This is why runForeach or runFold are your best friends. They process the element and then discard it, keeping the memory footprint constant regardless of whether you process 10 lines or 10 billion lines.

📋 Practical Task

Exercise: Building a Streaming Log-to-Metric Aggregator

You need to build a system that processes a stream of simulated log strings and calculates the total number of "WARN" and "ERROR" messages separately, but without loading the whole stream into memory.

Requirements:

• Create a ZStream from a List of at least 20 strings (some containing "INFO", "WARN", and "ERROR").
• Use a runFold to aggregate the results into a Map[String, Int] where the key is the log level and the value is the count.
• The stream should only count "WARN" and "ERROR" levels; "INFO" messages should be filtered out entirely before they reach the fold.
• Print the final Map to the console.