I’ve seen this pattern in almost every legacy codebase I’ve touched: a developer realizes that external APIs are flaky, so they wrap their background job logic in a begin...rescue block. When the API throws a 500 or a timeout, the developer simply tells the job to enqueue itself again. On the surface, it looks like you've solved the problem. The job didn't "fail," and the task eventually gets done. But in reality, you've just built a very efficient way to DDOS your own dependencies.

The "Infinite Loop" Trap

Let's look at a typical naive implementation. Imagine we have a job that syncs a user's subscription status to an external CRM like Salesforce. If the CRM is down, the naive approach looks something like this:

class SyncUserToCrmJob
  include Sidekiq::Job

  def perform(user_id)
    user = User.find(user_id)
    CrmClient.sync_user(user)
  rescue CrmClient::ApiError => e
    # "Oh no, it failed! I'll just try again immediately."
    SyncUserToCrmJob.perform_async(user_id)
  end
end

Here is why I'll stop you in a code review if I see this. First, you've bypassed the job framework's built-in tracking. To Sidekiq or ActiveJob, this job "succeeded" because you rescued the exception. You'll never see these failures in your monitoring dashboard. Second, you're retrying immediately. If the CRM is down for maintenance for ten minutes, this job will fire thousands of times per second, hammering a server that is already struggling to stay alive. It's a death spiral. You aren't giving the external service room to breathe, and you're wasting your own Redis memory and worker cycles.

Trusting the Queue and Embracing Backoff

The better way is to embrace the "fail fast" philosophy. Instead of hiding the error, you should let the exception bubble up. When a job raises an unhandled exception, the queue manager catches it and puts it into a retry set. The magic here is exponential backoff.

Most Ruby job libraries don't just retry immediately; they wait a few seconds, then a few minutes, then an hour. This gives the external API time to recover. Here is how that looks in a professional implementation:

class SyncUserToCrmJob
  include Sidekiq::Job
  
  # Sidekiq defaults to ~25 retries over 21 days, 
  # but we can be more explicit.
  sidekiq_options retry: 5 

  def perform(user_id)
    user = User.find(user_id)
    CrmClient.sync_user(user)
    # No rescue block here! Let the error happen.
  end
end

By removing the rescue block, you get several things for free. You get visibility—your error tracker (like Sentry or Honeybadger) will actually tell you when the CRM is failing. You get a controlled retry schedule. And you get a "Dead Letter Queue" (the Dead set), where jobs go after all retries are exhausted. This allows you to manually intervene or replay the jobs once the CRM is back online, rather than having them vanish into a void of recursive calls.

The Hidden Danger of Side Effects

There is a catch, though. Once you move to a retry-based system, you have to ensure your jobs are idempotent. This is a fancy way of saying that running the same job twice should have the same result as running it once.

If your job does two things—like charging a credit card and then updating a database—and it fails on the database update, the retry will attempt to charge the credit card again. That is a catastrophic failure. I always tell my juniors: if a job can be retried, it must be safe to run multiple times. You achieve this by checking the state of the world before acting. Instead of just calling charge!, you check if order.paid? first. Only then do you proceed. When you design for failure, you aren't just writing code for the "happy path"; you're writing code that can survive a chaotic production environment.

📋 Practical Task

Fixing the Webhook Delivery Death-Spiral

You have inherited a WebhookDeliveryJob that is causing spikes in CPU usage whenever a client's server goes offline. The current implementation uses a recursive rescue block that retries the job immediately.

Your Goal: Rewrite the job to remove the manual retry logic and instead leverage the framework's built-in retry mechanism. Additionally, implement a basic idempotency check to ensure that if a webhook was already marked as delivered in the database, the job exits early without making another network request.

# Current broken implementation
class WebhookDeliveryJob
  include Sidekiq::Job

  def perform(webhook_id)
    webhook = Webhook.find(webhook_id)
    
    begin
      HTTParty.post(webhook.url, body: webhook.payload)
      webhook.update!(delivered: true)
    rescue StandardError => e
      # This is the "death spiral" logic that needs to go
      WebhookDeliveryJob.perform_async(webhook_id)
    end
  end
end

Modify this class so that it:

1. Checks if webhook.delivered? is true and returns early if so.
2. Allows exceptions to bubble up to the queue manager.
3. Configures a maximum of 3 retries.