Kafka is often painted as this monolithic, intimidating beast used only by FAANG companies, but at its core, it's just a distributed commit log. For us in Ruby, it's a fantastic way to decouple services. Instead of your Rails app waiting for a slow third-party API to respond, you just throw a message into Kafka and let a background worker handle the heavy lifting.

To keep things concrete, let's build a small "Order Processing" pipeline for a fictional bakery. We need a way to signal that an order has been placed so that a separate notification service can send a confirmation email without slowing down the checkout process.

Setting up the Bakery Producer

First, we need a way to push data into Kafka. I'm using the ruby-kafka gem here because it's straightforward for learning the fundamentals. I want to create a simple script that simulates a customer clicking "Buy" on a sourdough loaf.

require 'kafka'

kafka = Kafka.new(["localhost:9092"], client_id: "bakery-checkout")

# We'll create a simple hash to represent our order
order_data = { order_id: 12345, item: "Sourdough Loaf", customer_email: "ruby@example.com" }.to_json

kafka.deliver_message(order_data, topic: "bakery_orders")
puts "Order sent to Kafka!"

In a real app, you wouldn't instantiate Kafka.new every time you send a message—that's a performance killer. You'd want a singleton or a long-lived connection. But for this script, it gets the job done.

Listening for Orders with a Consumer

Now we need a separate process that acts as our "Notification Service." This is the consumer. The key thing to remember here is the group_id. Kafka uses this to keep track of which messages this specific group of workers has already read. If you change the group ID, you can effectively "replay" the messages from the beginning.

require 'kafka'
require 'json'

kafka = Kafka.new(["localhost:9092"], client_id: "notification-service")

consumer = kafka.consumer(group_id: "bakery-notifier")
consumer.subscribe("bakery_orders")

puts "Waiting for orders..."

consumer.each_message do |message|
  order = JSON.parse(message.value)
  puts "Sending confirmation email to #{order['customer_email']} for order ##{order['order_id']}..."
  # In a real world, this is where your Mailer code goes
end

The "Where did my message go?" moment

While testing this, I ran into a classic Kafka gotcha. I decided to switch to an async_producer to make the checkout process feel snappier. I wrote a quick script, ran it, and... nothing happened. The consumer never saw the message.

Here is the mistake I made:

# MISTAKE: The script exits before the buffer flushes!
producer = kafka.async_producer
producer.produce(order_data, topic: "bakery_orders")
# Script ends here

Because the async_producer buffers messages in memory to send them in batches for efficiency, the script finished and the Ruby process exited before the producer had a chance to actually push the data to the broker. I felt like a beginner all over again.

The fix is simple: you must explicitly shut down the producer to ensure all buffered messages are flushed.

producer = kafka.async_producer
producer.produce(order_data, topic: "bakery_orders")
producer.shutdown # This forces the buffer to clear and send the data

Handling the stream in production

When you move this into a production environment, you'll find that running a raw each_message loop in a script is a bit fragile. You'll want to wrap your consumer in a proper process manager (like Systemd or Kubernetes) and implement a graceful shutdown. If you just kill -9 a Kafka consumer, you might end up with "rebalancing" delays where Kafka spends time figuring out who is still alive before it starts sending messages again.

I've found that for larger Ruby projects, the Karafka framework is the way to go—it handles the boilerplate of the consumer groups and threading for you—but understanding these raw ruby-kafka mechanics is essential before you let a framework hide the magic from you.

📋 Practical Task

Implement a Bakery Inventory Tracker

Your task is to build a second consumer that listens to the same bakery_orders topic. While the notification service sends emails, this new Inventory Tracker should simply print a message stating that the stock for the specific item has been decremented.

Requirements:

• Create a new Ruby script called inventory_tracker.rb.
• Use a different group_id than the "bakery-notifier" (e.g., "inventory-manager"). This is crucial; if you use the same group ID, Kafka will split the messages between the two consumers instead of sending every message to both.
• Parse the JSON message and output: "Inventory updated: [Item Name] stock decreased by 1."
• Run your producer script from the lesson, and verify that both the Notification Service and your new Inventory Tracker receive the message simultaneously.