A few years ago, I was working on a pricing engine that relied on a legacy SOAP service. To put it bluntly, the service was a disaster—it took nearly two seconds to respond to a single request. To save our sanity and the service's uptime, we implemented a caching layer. Initially, we just used a simple map. But we quickly hit a wall known as the "thundering herd" problem. When a high-traffic key expired, fifty concurrent requests would all notice the cache was empty at the same millisecond. All fifty would then hammer the legacy service simultaneously for the exact same data. We didn't just slow down the app; we effectively DDOS'd our own backend.

To fix this in a functional way using Cats Effect, we can't just use a synchronized block or a mutable map and hope for the best. We need to manage state atomically and handle the "pending" state of a request so that only one fiber ever performs the actual fetch.

Managing the Cache State with Ref

In Cats Effect, the go-to for shared, mutable state is Ref. For a cache, you might think a Ref[F, Map[K, V]] is enough. While that works for basic lookups, it doesn't solve the thundering herd. If you check the map, find it empty, and then call the API, other fibers will also find it empty before your first API call returns and updates the map.

The trick is to change what you store in the map. Instead of storing the value V, you store a Deferred[F, V]. A Deferred is essentially a one-time promise. When a fiber looks at the cache and sees no entry, it atomically creates a Deferred and puts it in the map. Any subsequent fiber that comes along will see that Deferred already exists and simply wait for it to be completed. I've found this pattern to be the cleanest way to ensure that no matter how many requests hit your service, the expensive computation only happens once.

Orchestrating the Fetch Logic

The implementation requires a bit of careful coordination. You need to use Ref.modify to ensure that the act of checking the map and inserting the Deferred happens in one atomic step. If you separate the "check" from the "insert," you've just reintroduced the race condition we're trying to avoid.

Once you have the Deferred, the logic splits: the "winner" (the fiber that created the Deferred) is responsible for calling the slow API and completing the Deferred. The "waiters" simply call deferred.get. It's important to wrap the actual API call in a handleErrorWith or similar mechanism; otherwise, if the API call fails, the Deferred will never be completed, and every fiber waiting on that key will hang forever. I always recommend putting a timeout on the API call or explicitly completing the Deferred with a failure to unblock the rest of the system.

// A conceptual sketch of the internal logic
def getOrElse(key: K, fetch: F[V]): F[V] = {
  cacheRef.modify { map =>
    map.get(key) match {
      case Some(deferred) => 
        // Someone else is already fetching this
        (map, deferred.get)
      case None => 
        // I am the winner, I must fetch
        val deferred = Deferred.newDeferred[F, V]
        val action = fetch.flatMap(v => deferred.complete(v).as(v))
        (map + (key -> deferred), action)
    }
  }.flatten
}


Notice how we return the F[V] (either the get or the action) inside the modify block and then flatten it. This keeps the critical section of the Ref extremely short, ensuring we don't block other keys from being accessed while one fiber is waiting for the network.

📋 Practical Task

Exercise: Building a Thundering-Herd-Proof API Cache

Your task is to implement a CachingLayer class that wraps an expensive asynchronous call and ensures that concurrent requests for the same key do not trigger multiple calls to the underlying service.

Requirements:

• Create a class CachingLayer[F[_]: Sync, K, V].
• It should maintain an internal state using Ref[F, Map[K, Deferred[F, V]]].
• Implement a method get(key: K, fetch: F[V]): F[V].
• If the key is already being fetched, subsequent calls must wait for the original Deferred to complete.
• If the key is not present, the first fiber must create the Deferred, perform the fetch, and then complete the Deferred.
• Ensure that if the fetch fails, the error is propagated to all waiting fibers (Hint: you may need to handle the failure to avoid leaving Deferreds hanging).

Starter Template:

import cats.effect._
import cats.effect.std.Deferred
import cats.syntax.all._

class CachingLayer[F[_]: Sync, K, V](state: Ref[F, Map[K, Deferred[F, V]]]) {
  def get(key: K, fetch: F[V]): F[V] = {
    ??? 
  }
}