When you're dealing with a monolith, a stack trace is your best friend. But once you move to microservices, that stack trace stops at the edge of the first service. If a request fails in the fourth service down the chain, you're left grepping logs across four different machines, trying to match timestamps and hoping your request_id was passed along correctly. That's why we use distributed tracing.

Today, we're going to build a tiny version of this. We'll create an OrderService that calls a PaymentService. We'll use OpenTelemetry (OTel), which is the industry standard right now, to make sure we can see the entire lifecycle of a request across both processes.

Setting the Stage with an Order and Payment Flow

I'm going to keep this simple. We aren't going to boot up full Rails apps; instead, we'll use two small Ruby scripts. One will act as the "frontend" (Order) and the other as the "backend" (Payment). For HTTP calls, I'm using Faraday because it's flexible and plays well with OTel middleware.

# Gemfile
gem 'opentelemetry-sdk'
gem 'opentelemetry-exporter-otlp'
gem 'faraday'
gem 'sinatra'

In a real production app, you'd send these traces to Jaeger or Honeycomb. For this lesson, I'm configuring the SDK to print to the console. It's not pretty, but it lets us verify the Trace IDs without needing to spin up a whole observability stack.

Wiring up the OpenTelemetry SDK

First, we need to initialize the tracer. I usually wrap this in a helper method so I don't clutter my main application logic. Notice how I'm naming the service—this is crucial because, in a sea of microservices, "ruby-app" is a useless name.

require 'opentelemetry/sdk'

OpenTelemetry::SDK.configure do |c|
  c.service_name = 'order-service'
  # In a real world scenario, we'd use the OTLP exporter here.
  # For now, we'll just log to stdout to see the magic happen.
end

tracer = OpenTelemetry.tracer_provider.tracer('order-tracer')

Now, let's wrap our "place order" logic in a span. A span represents a single unit of work. If the whole request is a "Trace," the span is a "Step" within that trace.

tracer.in_span('process_order') do |span|
  span.set_attribute('order.id', '12345')
  puts "Order service is processing..."
  
  # Call the Payment service
  Faraday.get('http://localhost:4567/pay')
end

The "Silent Failure" of Broken Traces

Here is where I almost tripped up. I wrote the PaymentService with the exact same SDK configuration as the OrderService. I ran both, made a request, and looked at the output. I saw a span for process_order and a span for process_payment, but they had different Trace IDs.

This is a classic mistake. I had instrumented both services, but I hadn't propagated the context. The PaymentService had no idea it was being called as part of an existing trace, so it just started a brand new one. In a production environment, this means your distributed trace is broken, and you're back to square one—searching logs manually.

Connecting the Dots with Context Propagation

To fix this, we need to inject the trace context into the HTTP headers of the request and then extract it on the receiving end. OpenTelemetry provides "propagators" to handle this automatically.

In the OrderService, I need to make sure the headers are injected. While the OTel Faraday instrumentation usually handles this, doing it manually once helps you understand what's actually happening under the hood:

# Injecting the current span context into the headers
headers = {}
OpenTelemetry.propagation.inject(headers)
Faraday.get('http://localhost:4567/pay', nil, headers)

Now, on the PaymentService side (which I'm running via a simple Sinatra app), I need to extract that context before starting the span. If I don't extract it, the PaymentService won't know it's a child of the OrderService span.

# Payment Service (Sinatra)
get '/pay' do
  # Extract the context from the request headers
  parent_context = OpenTelemetry.propagation.extract(request.env)
  
  # Wrap the work in a span, explicitly using the extracted context
  OpenTelemetry::Context.with_current(parent_context) do
    tracer.in_span('process_payment') do |span|
      span.set_attribute('payment.amount', 100)
      "Payment Successful"
    end
  end
end

Now, when you run this, both spans will share the same trace_id. You can finally see the request travel from Order to Payment as one continuous timeline. It's a small change in code, but it's the difference between knowing exactly why a request took 5 seconds and guessing which service was the bottleneck.

📋 Practical Task

Implement Trace Propagation for a Shipping Service

You have a system where OrderService calls PaymentService, and PaymentService then calls a ShippingService to arrange delivery. You have successfully implemented tracing between Order and Payment, but the ShippingService is currently "dark"—it creates its own independent traces, making it impossible to track a single order from start to finish.

Your task: Modify the ShippingService (a Sinatra application) to extract the trace context from the incoming request headers and wrap its shipping logic in a span that is linked to the parent trace from the PaymentService.

Requirements:

• Use OpenTelemetry.propagation.extract to retrieve the context from the request.
• Ensure the in_span('arrange_shipping') block is executed within the extracted context.
• Add a span attribute shipping.carrier with the value "FedEx" to the shipping span.