We’ve all been there. You launch an A/B test—maybe you're testing a new landing page layout to increase sign-ups—and you've calculated that you need 5,000 users per variant to reach statistical power. But you're anxious. You start checking the results every morning. On day three, you see a p-value of 0.04. You're tempted to stop the test right there, declare victory, and ship the feature. It feels efficient, right? Why waste another 3,000 users when the "math" says you're already done?

The Danger of the Daily Peek

Here is the hard truth: if you check your p-value repeatedly and stop the moment it hits 0.05, you aren't actually testing at a 5% significance level. You are "peeking," and peeking is a silent killer of statistical validity. When you check the data multiple times, you're essentially giving the noise multiple chances to look like a signal. In a random walk, the probability that the p-value will eventually dip below 0.05 is much higher than 5%, even if there is absolutely no difference between your variants.

In R, the naive approach looks like this. You might write a loop that simulates data coming in and runs a standard prop.test() at every interval:

# The WRONG way: Peeking at data
set.seed(123)
n_total <- 5000
successes_a <- rbinom(n_total, 1, 0.10)
successes_b <- rbinom(n_total, 1, 0.10) # No real difference

for (i in seq(100, n_total, by = 100)) {
  p_val <- prop.test(sum(successes_a[1:i]), sum(successes_b[1:i]), 
                     n = c(i, i))$p.value
  if (p_val < 0.05) {
    print(paste("Stop at n =", i, "Significant!"))
    break
  }
}

If you run a simulation of that loop thousands of times, you'll find that your False Positive Rate isn't 5%—it's often 20% or 30% depending on how often you peek. You've traded scientific rigor for a feeling of speed, and you'll end up shipping features that don't actually work.

Trading Certainty for Flexibility

To do this properly, we need a Sequential Testing method. The goal is to allow for early stopping while keeping the overall Type I error (the chance of a false positive) capped at 0.05 across the entire experiment. The trade-off is that the "barrier" for significance becomes much harder to hit early on. You can't just use 0.05; you have to use a boundary that adjusts based on how many times you've looked at the data.

One of the most respected methods for this is the O'Brien-Fleming boundary. It's intentionally conservative at the start—meaning you need an incredibly strong result to stop early—and becomes more lenient as you approach your maximum sample size. It's like telling yourself, "I'll only stop on day three if the result is so obvious it's undeniable; otherwise, I'll wait until the end."

Implementing Group Sequential Design

In R, the gsDesign package is the industry standard for this. Instead of guessing your boundaries, you define your desired power, your alpha, and how many "looks" (interim analyses) you plan to take. The package then calculates the exact p-value thresholds for each look.

library(gsDesign)

# Design a study with 4 looks (interim analyses)
# Alpha = 0.05, Power = 0.8, 4 looks, O'Brien-Fleming boundary
design <- gsDesign(k = 4, 
                   test.type = 2, 
                   alpha = 0.05, 
                   beta = 0.2, 
                   sfu = sfLDOF)

# Look at the boundaries for the Z-score
print(design$upper$bound)

When you run this, you'll notice the first boundary is very high (e.g., a Z-score of 4.0), while the final boundary is much closer to the traditional 1.96. I prefer this approach because it removes the emotional temptation to stop early. You have a pre-defined "rulebook." If the data hits the boundary, you stop. If it doesn't, you keep collecting. You get the best of both worlds: the ability to end a landslide victory early and the mathematical certainty that you aren't just chasing ghosts in the noise.

📋 Practical Task

Building an O'Brien-Fleming Boundary Monitor

You are tasked with creating a monitoring function for a high-stakes A/B test. Instead of peeking naively, you must implement a check that compares the current Z-score of a test against a pre-defined O'Brien-Fleming boundary.

Requirements:

• Load the gsDesign package.
• Create a gsDesign object for a test with 5 planned looks (interim analyses), an alpha of 0.05, and a beta of 0.2.
• Write a function called check_early_stop(current_look, current_z) that:
  • Takes the look number (1 through 5) and the current calculated Z-score.
  • Returns TRUE if the absolute value of the Z-score exceeds the O'Brien-Fleming boundary for that specific look.
  • Returns FALSE otherwise.
• Test your function with a Z-score of 3.0 at look 1 (should be FALSE) and a Z-score of 2.1 at look 5 (should be TRUE).