You've likely spent a lot of time with p-values and confidence intervals. They're the industry standard, but if you've ever tried to explain a p-value to a product manager, you know it's a nightmare. They don't want to know "the probability of seeing this result given the null hypothesis is true"; they want to know "What is the probability that Version B is actually better than Version A?"

That's where Bayesian A/B testing comes in. Instead of a binary "significant or not," we treat the conversion rate as a distribution of probabilities. For conversion rates (binary outcomes), the Beta distribution is our best friend because it's the conjugate prior to the Binomial distribution. In plain English: if your data is "success or failure," the Beta distribution is the mathematically perfect way to describe your uncertainty about the success rate.

Defining our priors and observing the data

Let's say we're testing a new checkout button color. We've got a Control group (green) and a Variant group (blue). I'll start by defining my observed data. We'll keep it simple: 1,000 visitors per group.

# Observed data
control_conv <- 50
control_total <- 1000

variant_conv <- 65
variant_total <- 1000

In Bayesian land, we start with a "prior"—what we believe about the conversion rate before seeing any data. A common choice is the "flat prior" (Beta(1,1)), which basically says "I have no idea what's going to happen; any rate from 0 to 1 is equally likely."

Simulating the posterior distributions

Since calculating the exact integral for the probability that B > A can be a headache, we use simulation. We draw thousands of random samples from the posterior distributions of both groups and simply count how often B is larger than A.

# Setting a seed for reproducibility
set.seed(42)
n_samples <- 100000

# Posterior for Control: Beta(prior_alpha + successes, prior_beta + failures)
# I'll use a prior of Beta(1, 1)
control_posterior <- rbeta(n_samples, 1 + control_conv, 1 + (control_total - control_conv))

# Posterior for Variant
variant_posterior <- rbeta(n_samples, 1 + variant_conv, 1 + (variant_total - variant_conv))

Wait, my prior is pulling the data too hard

Here is where I usually trip up when I'm trying to be "helpful" with my priors. A few minutes ago, I actually tried using a "strong prior" because I knew our historical conversion rate was around 5%. I used Beta(50, 950). But when I ran the simulation, I noticed the Variant's lift was being completely dampened. I was essentially telling the model, "I am so sure the rate is 5% that it takes a mountain of evidence to move the needle."

In most A/B tests, unless you have a very specific reason to bake in historical data, stick to the uninformative Beta(1,1). It lets the current experiment speak for itself. I've reverted the code above to use 1 for the alpha and beta parameters to avoid that "anchor" effect.

Calculating the probability of victory

Now for the part the stakeholders actually care about. We compare the two vectors of samples. If variant_posterior[i] > control_posterior[i], the variant won that specific simulation.

# Calculate the probability that Variant is better than Control
prob_B_better_than_A <- mean(variant_posterior > control_posterior)

print(paste0("Probability that Variant is better: ", round(prob_B_better_than_A * 100, 2), "%"))

# Let's also look at the expected lift
lift <- (variant_posterior / control_posterior) - 1
mean_lift <- mean(lift)
print(paste0("Expected lift: ", round(mean_lift * 100, 2), "%"))

This output is gold. Instead of saying "p = 0.12, so we fail to reject the null," you can say, "There is an 88% chance that the blue button is better than the green button, with an expected lift of 30%." It's a completely different conversation, and usually a much more productive one.

📋 Practical Task

Implement a Bayesian Comparison for a Sign-up Flow

You are tasked with analyzing a sign-up flow optimization. You have the following data:

• Control: 2,500 visitors, 120 conversions.
• Variant: 2,500 visitors, 155 conversions.

Write an R script that performs the following:

1. Defines the observed successes and totals for both groups.
2. Uses a flat prior (Beta(1,1)) to simulate 100,000 samples for both the control and variant posterior distributions.
3. Calculates and prints the probability that the Variant conversion rate is higher than the Control conversion rate.
4. Calculates the 95% Credible Interval for the lift (the difference between the variant rate and control rate). Hint: Use the quantile() function on the difference between the two posterior samples.