Up until now, you've probably been running Sidekiq with the default settings. For a few background jobs, the defaults are fine. But once you start hitting production-level traffic or dealing with resource-heavy tasks, "default" becomes a dangerous word. Today, we're going to look at how to actually tune your Sidekiq configuration to balance throughput with system stability.

Let's imagine we're building a feature for a social app where users upload high-resolution profile pictures. We need to generate three different sizes (thumbnail, medium, and large) for every upload. This is a classic "heavy" job—it's CPU intensive and takes a few seconds per image.

Defining the Image Processor

First, let's get our worker in place. I'm keeping this simple; I'm assuming you have a library like mini_magick installed to handle the actual resizing.

class ImageProcessorWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'images', retry: 3

  def perform(user_id, image_path)
    user = User.find(user_id)
    # Imagine some heavy image manipulation happening here
    ImageService.resize_all_versions(image_path)
    user.update!(profile_processed: true)
  end
end

At this point, if you run sidekiq from the terminal, it defaults to 25 concurrency threads. That means one Sidekiq process will try to process 25 images simultaneously. On a powerful server, that's great. On a small Heroku dyno or a cheap VPS, your CPU will spike to 100%, and your app will crawl.

Moving Configuration to YAML

I hate passing flags like -c 10 -q images -q default in the terminal. It's brittle and easy to forget. Instead, let's create a config/sidekiq.yml file. This is where we can formally define our queues and how the process should behave.

# config/sidekiq.yml
:concurrency: 10
:queues:
  - [images, 2]
  - [default, 1]

Notice the weights I added to the queues. By putting [images, 2], I'm telling Sidekiq that the images queue is twice as important as the default queue. It doesn't mean it processes two jobs at once; it means Sidekiq is more likely to check the images queue first. I prefer this over strict ordering because it prevents the default queue from being completely starved of attention if a million images are uploaded at once.

The "More is Better" Trap

Here is where I usually trip up when I'm rushing a project. I remember thinking, "We have a massive backlog of images, let's just crank the concurrency up to 50 and blast through them." I updated my sidekiq.yml to :concurrency: 50 and restarted the process.

Almost immediately, the logs started screaming with ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds.

I forgot the Golden Rule of Sidekiq concurrency: Your Sidekiq concurrency must be less than or equal to your database connection pool size.

Each Sidekiq thread is like a separate worker. If you have 50 threads all trying to talk to PostgreSQL at the same time, but your database.yml only allows 5 connections, 45 of those threads are just sitting there waiting for a connection until they eventually time out and crash the job.

Aligning the Pool and the Process

To fix this, I have to make sure my database configuration can handle the concurrency I've requested. If I want 20 threads to process images, I need at least 20 connections available in the pool.

# config/database.yml
production:
  adapter: postgresql
  pool: 25 # I give it a little extra breathing room

Now, we match that back in our Sidekiq config:

# config/sidekiq.yml
:concurrency: 20
:queues:
  - [images, 2]
  - [default, 1]

Now the system is balanced. We aren't overloading the CPU with 50 simultaneous image resizes, and we aren't choking the database by asking for more connections than we've allocated. When you're tuning this in your own apps, start low. It's much easier to increase concurrency and watch your CPU usage climb than it is to debug random connection timeouts in a production environment.

📋 Practical Task

Exercise: Resolving the Connection Bottleneck

You have been handed a legacy Ruby project. The application is crashing under load. You find the following configurations:

# config/database.yml
production:
  adapter: mysql2
  pool: 5

# config/sidekiq.yml
:concurrency: 15
:queues:
  - default

The logs are filled with ActiveRecord::ConnectionTimeoutError. Your task is to modify both files to support a concurrency of 15 workers while ensuring the database pool is sufficient to prevent timeouts, leaving a buffer of 2 extra connections for safety. Rewrite the pool value in database.yml and the :concurrency value in sidekiq.yml to resolve the issue.