I've seen this happen in production more times than I'd like to admit: a system that looks perfect during happy-path testing, but leaves a trail of "zombie data" the moment a network timeout hits. In a monolithic app, you just wrap everything in a database transaction and call commit() or rollback(). But in a distributed system—where your payment logic is in one service and your inventory in another—you can't just lock two different databases across the network. It's too slow and far too fragile.

The "Half-Baked" Order Problem

Imagine we're building a travel booking system. We need to charge the customer's card and then reserve a hotel room. A junior dev on my team once wrote something like this. It looks clean, right?

case class Booking(id: String, userId: String, amount: Double, hotelId: String)

def processBooking(booking: Booking): Future[Unit] = {
  for {
    _ <- paymentService.charge(booking.userId, booking.amount) // Step 1
    _ <- hotelService.reserveRoom(booking.hotelId)            // Step 2
    _ <- bookingService.markAsConfirmed(booking.id)           // Step 3
  } yield ()
}.recover {
  case ex: Exception => 
    println(s"Booking failed: ${ex.getMessage}")
    // We log the error, but the money is already gone!
}

Here is the disaster: if hotelService.reserveRoom fails (maybe the room was taken a millisecond ago), the paymentService.charge has already happened. The customer has been charged, but they have no hotel room and no confirmed booking. The money is essentially stolen by the system. You can't just "undo" a network call to a third-party payment gateway with a catch block.

Implementing Compensating Transactions

To fix this, we use the Saga Pattern. Instead of trying to prevent failure (which is impossible in distributed systems), we accept that failures will happen and define a "compensating transaction" for every action we take. Think of it as a programmatic "Undo" button.

I prefer the Orchestrator approach for clarity. We create a coordinator that knows exactly how to roll back the sequence based on how far the process got.

trait SagaStep {
  def execute(): Future[Unit]
  def compensate(): Future[Unit]
}

def processBookingSaga(booking: Booking): Future[Unit] = {
  val steps = List(
    new SagaStep {
      def execute() = paymentService.charge(booking.userId, booking.amount)
      def compensate() = paymentService.refund(booking.userId, booking.amount)
    },
    new SagaStep {
      def execute() = hotelService.reserveRoom(booking.hotelId)
      def compensate() = hotelService.cancelReservation(booking.hotelId)
    },
    new SagaStep {
      def execute() = bookingService.markAsConfirmed(booking.id)
      def compensate() = bookingService.markAsFailed(booking.id)
    }
  )

  def runSaga(remainingSteps: List[SagaStep], completedSteps: List[SagaStep]): Future[Unit] = {
    remainingSteps match {
      case Nil => Future.successful(()) // All steps succeeded!
      case head :: tail => 
        head.execute().flatMap { _ => 
          runSaga(tail, head :: completedSteps) 
        }.recoverWith {
          case ex => 
            println(s"Failure detected: ${ex.getMessage}. Starting compensation...")
            rollback(completedSteps).flatMap(_ => Future.failed(ex))
        }
    }
  }

  def rollback(completedSteps: List[SagaStep]): Future[Unit] = {
    // Roll back in reverse order of execution
    val rollbackFutures = completedSteps.map(_.compensate())
    Future.sequence(rollbackFutures).map(_ => ())
  }

  runSaga(steps, Nil)
}

Why this works is that we've shifted from "Atomic Transactions" (all or nothing right now) to "Eventual Consistency." If the hotel reservation fails, the rollback method kicks in and triggers the refund. The user might see a charge and a refund on their statement a few seconds apart, but the system remains consistent.

The Reality of "Failure to Undo"

Now, here is a professional tip: the Saga pattern isn't a magic bullet. What happens if the compensate() call itself fails? If the refund service is down, you're back to square one.

In a production-grade Saga, you can't just rely on in-memory Future chains. You need a persistent log (often called a Saga Log or State Machine) that records every step. If the server crashes mid-rollback, a background worker reads the log and says, "Wait, we charged this user but never successfully refunded them," and retries the compensation until it succeeds. You move from "trying to fix it once" to "guaranteeing it eventually gets fixed."

📋 Practical Task

Exercise: Implementing a Distributed Flight and Car Rental Saga

You are building a travel package system. To complete a "Vacation Package," you must successfully call three services in order: FlightService, CarRentalService, and HotelService.

Requirements:

• Create a SagaStep trait with execute() and compensate() methods returning Future[Unit].
• Implement three concrete steps:
  • Flight: bookFlight() / cancelFlight()
  • Car: rentCar() / cancelCarRental()
  • Hotel: bookHotel() / cancelHotel()
• Write an orchestrator function bookVacationPackage(packageId: String) that executes these steps sequentially.
• If any step fails, all previously successful steps must be compensated in reverse order.
• Ensure that if the HotelService fails, the car rental is cancelled first, followed by the flight.

Test Case: Mock the CarRentalService to throw a RuntimeException("No cars available") and verify that FlightService.cancelFlight() is called, but HotelService.cancelHotel() is NOT called (since the hotel was never booked).