A few years ago, I was mentoring a junior analyst who was absolutely convinced that a new UI change had tanked our conversion rate. He had run a small pilot test with a few hundred users, saw a 2% dip, and was already drafting the "rollback" email to the product manager. I asked him if he'd checked the sampling distribution. He looked at me like I was speaking Greek. I sat him down and spent an hour writing a quick simulation in R to show him that, given the sample size and the baseline variance, a 2% dip would happen by pure chance about 12% of the time. He didn't send that email. That's the power of simulation: it turns "I think this is a trend" into "I know this is noise."

Defining the Ground Truth and the Sampling Loop

When you build a simulation, you aren't analyzing existing data; you're creating a synthetic universe where you already know the answer. This is what we call the "ground truth." In R, the most efficient way to handle this is by wrapping your experiment into a single function and then calling that function thousands of times. While you could use a for loop, I almost always recommend replicate() for these kinds of studies. It's cleaner and more idiomatic for statistical work.

For example, if we want to see how a sample mean behaves, we don't just take one sample. We define the true population mean (our ground truth), write a function that draws a random sample and calculates the mean, and then replicate that process. I've seen developers try to pre-allocate massive matrices to hold every single draw, but usually, you only care about the final statistic of each trial. Keep your function lean—only return the value you're actually testing.

# A simple example of the pattern
true_mean <- 50
run_trial <- function(n = 30) {
  sample <- rnorm(n, mean = true_mean, sd = 10)
  return(mean(sample))
}

# Run 10,000 simulations
results <- replicate(10000, run_trial())


Evaluating Bias and Variance in the Distribution
Once you have your results vector, you're no longer looking at a single data point, but a sampling distribution. This is where the real insight happens. I always tell my team to look for two things: bias and spread. Bias is simply the difference between the average of your simulation results and your ground truth. If your simulation average is 49.98 when the truth was 50, you're probably fine. If it's 45, your estimator is biased, and you've got a problem with your logic.

The spread—or the variance—tells you how much you can trust a single experiment. By plotting a histogram of your results, you can visually inspect the "margin of error." If the distribution is wide, your sample size (n) is too small to provide a stable estimate. You can then iterate on the simulation, increasing n until the distribution narrows to an acceptable level. It's a fast, iterative feedback loop that saves you from making expensive mistakes in production.

📋 Practical Task

Simulating the Power of a Conversion Rate A/B Test

In this exercise, you will build a simulation to determine how often a sample size of 500 users is sufficient to detect a real increase in conversion rates. This will help you understand "statistical power"—the probability that you'll actually detect an effect that exists.

Requirements:

• Define a ground truth: The Control group has a conversion rate of 10% (0.10) and the Treatment group has a conversion rate of 12% (0.12).
• Create a function called simulate_ab_test that:
  • Generates 500 random binary outcomes (0 or 1) for both groups using rbinom().
  • Calculates the difference in means (Treatment mean minus Control mean).
  • Returns that difference.
• Use replicate() to run this simulation 10,000 times.
• Calculate what percentage of those 10,000 trials resulted in a positive difference (greater than 0). This is your estimated power.
• Create a histogram of the differences to visualize the overlap between the two groups.