I’ve seen this exact scenario play out in a dozen different sprints. A developer pushes a new UI layout, runs a quick A/B test for a few hours, looks at the raw percentages, and tells the Product Manager, "The new design is winning by 2%! Let's ship it."

Here is what that "winning" logic usually looks like in code:

# A naive way to decide a winner
def determine_winner(control, treatment)
  control_rate = control[:conversions].to_f / control[:visitors]
  treatment_rate = treatment[:conversions].to_f / treatment[:visitors]

  puts "Control: #{(control_rate * 100).round(2)}%"
  puts "Treatment: #{(treatment_rate * 100).round(2)}%"

  treatment_rate > control_rate ? "Treatment Wins!" : "Control Wins!"
end

# Data from a small sample
control_group = { visitors: 100, conversions: 5 }    # 5%
treatment_group = { visitors: 100, conversions: 7 }  # 7%

puts determine_winner(control_group, treatment_group)
# Output: Treatment Wins!

The Trap of Raw Percentages

On the surface, the code works. It calculates the rates and picks the higher one. But as an engineer, you can't trust these results because the sample size is tiny. A difference of two conversions in a group of 100 people is almost certainly noise—random chance—rather than a result of your design change.

If you ship this, you're not making a data-driven decision; you're gambling. The "bug" here isn't a syntax error; it's a methodological error. We are ignoring statistical significance. We need to know if the observed difference is likely to persist over a larger population or if it's just a fluke of the small sample we happened to collect.

Calculating the P-Value via Z-Score

To fix this, we need to move beyond simple division and calculate a p-value. The p-value tells us the probability that the difference we're seeing happened by chance. Generally, in software experimentation, we look for a p-value of less than 0.05 (meaning there's less than a 5% chance the result is a fluke) before we call it a "win."

While there are gems like statsample for this, it's helpful to understand the math. We'll use a Z-test for proportions. I've rewritten the logic below to calculate the standard error and the resulting Z-score.

def significant_win?(control, treatment, confidence_level = 1.96)
  p_control = control[:conversions].to_f / control[:visitors]
  p_treatment = treatment[:conversions].to_f / treatment[:visitors]
  
  # Pooled probability
  p_pooled = (control[:conversions] + treatment[:conversions]).to_f / 
             (control[:visitors] + treatment[:visitors])
  
  # Standard Error
  se = Math.sqrt(p_pooled * (1 - p_pooled) * (1.0/control[:visitors] + 1.0/treatment[:visitors]))
  
  # Z-score: how many standard deviations the difference is from zero
  z_score = (p_treatment - p_control).abs / se
  
  # 1.96 is the Z-score threshold for 95% confidence
  z_score > confidence_level
end

control_group = { visitors: 100, conversions: 5 }
treatment_group = { visitors: 100, conversions: 7 }

if significant_win?(control_group, treatment_group)
  puts "Statistically significant result!"
else
  puts "Not enough evidence. Keep the test running."
end
# Output: Not enough evidence. Keep the test running.

Notice the difference. With the same data, the code now tells us the truth: 7% vs 5% with only 100 people per group isn't enough to prove anything. (If you bump those visitor counts to 10,000 each, the same percentage difference suddenly becomes highly significant).

Handling the "Peeking" Problem

One last piece of advice: don't run this script every five minutes and stop the test the moment the p-value hits 0.04. This is called "peeking," and it's a great way to accidentally find a false positive. Decide on your sample size before you start the experiment, or use a sequential analysis framework. Trust the math, but be wary of your own desire to see a "win" as quickly as possible.

📋 Practical Task

Build a Feature-Flag Experiment Validator

You are tasked with building a validation tool for your company's feature-flagging system. Create a Ruby class called ExperimentValidator that takes two datasets (control and treatment) and provides a detailed report on the results.

Your class should implement the following:

• A method calculate_lift that returns the percentage increase (or decrease) of the treatment conversion rate relative to the control rate.
• A method significant? that returns a boolean indicating if the result is statistically significant at a 95% confidence level (Z-score > 1.96).
• A method report that prints a summary including the conversion rates, the lift, and whether the result is significant enough to warrant a full rollout.

Test your class with these two scenarios:

1. Control: 1000 visitors, 50 conversions | Treatment: 1000 visitors, 70 conversions.
2. Control: 100 visitors, 5 conversions | Treatment: 100 visitors, 7 conversions.

Ensure your report clearly distinguishes between a "nominal win" (higher rate) and a "statistically significant win."