Scala
Completed
-
Section 1: Getting Started
-
Section 2: Core Syntax
-
Section 3: Object-Oriented Scala
-
Section 4: Functional Scala
-
Section 5: Collections in Depth
-
Section 6: Type System
-
Section 7: Concurrency and Ecosystem
-
Section 8: Practical Projects
-
Section 9: Interview Practice
-
Section 10: Data Structures and Algorithms in Scala
-
Section 11: More Practice Exercises
-
Section 12: Advanced Functional Patterns
-
Section 13: More Ecosystem
-
Section 14: Scala Collections Library Deep Dive
-
Section 15: Scala Standard Library Deep Dive
-
Section 16: Akka Ecosystem Deep Dive
-
Section 17: Cats and Cats Effect Deep Dive
-
Section 18: Play Framework Deep Dive
-
Section 19: Apache Spark with Scala Deep Dive
-
Section 20: Scala Build Tools Deep Dive
-
Section 21: Scala 3 Specific Features
-
90: Union and Intersection Types
-
Section 22: Scala Testing Deep Dive
-
Section 23: Functional Domain Modeling
-
Section 24: More Data Structures and Algorithms in Scala
-
Section 25: Scala for Data Engineering
-
Section 26: More Practical Projects
-
Section 27: More Interview and Review
-
Section 28: ZIO Ecosystem Deep Dive
-
Section 29: Scala for Machine Learning
-
Section 30: Scala Microservices Architecture
-
Section 31: Scala Type System Deep Dive
-
Section 32: More Practice and Drills
-
Section 33: Scala Performance Deep Dive
-
Section 34: Scala Ecosystem Tooling
-
Section 35: Scala for Reactive Systems
-
Section 36: More Real-World Case Studies
-
Section 37: Scala for Financial Systems
-
Section 38: Scala GraphQL and gRPC
-
Section 39: More Final Projects
-
Section 40: More Interview and Final Review
-
Section 41: Scala for Streaming Data
-
Section 42: Scala Security Practices
-
Section 43: More Language Deep Dive
-
Section 44: Scala Command-Line Tools
-
Section 45: Scala Documentation and Style
-
Section 46: Scala Dependency Management
-
Section 47: More Practical Backend Patterns
-
Section 48: Scala for Event-Driven Architecture
-
Section 49: More Practice Drills Round 2
-
Section 50: Scala Compiler Deep Dive
-
Section 51: Scala for Web Frontends
-
Section 52: More Data Engineering Practice
-
Section 53: Scala Observability
-
Section 54: More Advanced Practice Projects
-
Section 55: Scala for Legacy Java Integration
-
Section 56: More Testing Practice
-
Section 57: Final Mastery Review
-
Section 58: Scala History and Ecosystem Context
-
Section 59: More Concurrency Patterns
-
Section 60: Scala for Configuration Management
-
Section 61: More Domain Modeling Practice
148: Backpressure Handling in Reactive Streams
By the time you hit lesson 148, you've probably realized that "asynchronous" doesn't mean "magic." In a perfect world, your data sources and your data sinks move at the exact same speed. In the real world, your producer is often a firehose and your consumer is a straw. That's where backpressure comes in.
Wait, if the stream is asynchronous, how does the consumer actually tell the producer to slow down?
This is the part that trips people up. Most people think of streams as "pushing" data—the source just screams data at the sink until the sink crashes. But Reactive Streams (and by extension, Akka or Pekko Streams) use a pull-based model.
Think of it as a contract. The sink sends a "request" upstream saying, "I can handle 5 more elements." The producer is forbidden from sending more than those 5. Once the sink processes a few, it sends another request for more. I like to think of it as a conveyor belt where the person at the end of the line has a remote control to speed up or slow down the belt based on how full their workbench is.
// A simple conceptual look at a slow sink val slowSink = Sink.foreach[String] { element => Thread.sleep(100) // Simulate heavy work, like a DB write println(s"Processed $element") } // The source won't just dump 1 million elements into memory. // It will only emit what the slowSink has requested. Source(1 to 1000000).map(_.toString).runWith(slowSink)What if I'm dealing with a source I can't control, like a TCP socket or a sensor?
Here is the cold truth: you can't always tell the "internet" or a hardware sensor to slow down. If the data keeps coming and your sink is too slow, the data has to go somewhere. If you just let it pile up in memory, you'll get an
OutOfMemoryError, and your app will die. This is where you have to make a conscious business decision about overflow strategies.You generally have three choices: buffer it (risk memory), drop it (risk data loss), or fail (risk availability). If I'm building a real-time dashboard for stock prices, I don't care about the price from 10 seconds ago if I have the price from 1 second ago. In that case, I'll use
dropOldest.import org.apache.pekko.stream.OverflowStrategy // We create a buffer of 100 elements. // If the sink is too slow and the buffer fills up, we toss the oldest data. val stream = Source(sensorData) .buffer(100, OverflowStrategy.dropOldest) .via(heavyProcessingStage) .to(Sink.ignore)I've seen developers default to
OverflowStrategy.backpressure(the default) in scenarios where the source was a UDP stream. It doesn't work. UDP doesn't care about your backpressure; it just keeps hitting your network card. In those cases, you must define a dropping strategy or a bounded buffer.How do I actually prove that backpressure is working in my app?
It's hard to "see" backpressure because when it's working, everything just looks... slow. To verify it, I usually introduce an artificial bottleneck and monitor the buffer levels or use a
log()stage. If you see the logs from your source stop firing while the sink is still chugging away, you've successfully triggered backpressure.If you want to be scientific about it, you can use
monitorstages or custom probes, but for 90% of my debugging, I just add adelayto the sink and watch the source's behavior. If the source is a file reader, you'll notice the file pointer isn't moving forward as fast as the CPU allows—that's the pull-model in action.import scala.concurrent.duration._ Source(1 to 100) .log("Source") // Watch these logs .delay(10.milliseconds) .via(Flow[Int].map { x => Thread.sleep(50) // Artificial bottleneck x * 2 }) .log("Sink") .runWith(Sink.ignore)When you run this, you'll notice the "Source" logs happen in bursts, then pause. The source is waiting for the sink to say, "Okay, I'm ready for the next batch."
📋 Practical Task
Exercise: Implementing a Lossy Telemetry Pipeline
You are building a telemetry system for a fleet of drones. The drones send altitude data every 10ms, but your database can only write one record every 100ms. You cannot stop the drones from sending data, and you've decided that keeping the most recent data is more important than keeping every single data point.
Your Task:
- Create a
Sourcethat emits a sequence of integers (representing altitude) from 1 to 1,000. - Introduce a
bufferstage with a capacity of 10 elements. - Configure the buffer to use an
OverflowStrategythat discards the oldest elements when the buffer is full. - Create a
Sinkthat simulates a slow database write by usingThread.sleep(100). - Run the stream and verify (via print statements) that the sink is receiving numbers that "jump" (e.g., 1, 2, 15, 26...), proving that intermediate values were dropped due to backpressure.
There are no comments for now.