I've seen this mistake countless times in production environments: a data scientist or researcher runs a power analysis, gets a result that says they only need 12 samples to prove their hypothesis, and proceeds to spend a month collecting data—only to realize later that their study was hopelessly underpowered. The problem usually isn't the R code itself, but a fundamental misunderstanding of what the function expects as an input.

library(pwr)

# Scenario: We are testing a new feature to increase average order value (AOV).
# Current AOV is $50. We hope the new feature pushes it to $52.
# The raw difference is $2.

# I'll just plug the difference into the d parameter
result <- pwr.t.test(d = 2, 
                     sig.level = 0.05, 
                     power = 0.8, 
                     type = "two.sample")

print(result)
# Output suggests a sample size of ~3 per group. 
# That can't be right.

Mistaking Raw Difference for Effect Size

If you ran the code above, R didn't throw an error. That's the danger. The pwr.t.test function is doing exactly what you told it to do: it's assuming that the "Effect Size" (d) is 2. In the world of Cohen's d, an effect size of 2.0 is absolutely gargantuan—it's like claiming a drug doesn't just lower blood pressure, but practically cures the condition entirely. Because the effect is so massive, the math says you only need a handful of people to see it.

The bug here is that the user passed the raw difference in means ($2) instead of the standardized difference. Power analysis doesn't care about the currency or the units; it cares about how many standard deviations the means are apart. If your AOV has a standard deviation of $20, a $2 increase is a tiny ripple in the noise, not a mountain.

Calculating Cohen's d for Realistic Results

To fix this, you have to calculate the effect size by dividing the difference in means by the pooled standard deviation. If we assume our historical data shows a standard deviation of roughly $20, our real d is $2 / 20 = 0.1$.

library(pwr)

# Define our parameters
mean_diff <- 2
std_dev <- 20
calculated_d <- mean_diff / std_dev # This is Cohen's d

# Now run the power analysis with the standardized effect
result <- pwr.t.test(d = calculated_d, 
                     sig.level = 0.05, 
                     power = 0.8, 
                     type = "two.sample")

print(result)
# Now we see a sample size of ~392 per group. 
# This is a much more realistic requirement for a 4% lift.

By standardizing the input, we've moved from a delusional sample size to a scientifically sound one. I always recommend calculating your d explicitly in a variable rather than hard-coding it; it makes your intent clear to anyone reviewing your script.

Tuning the Power-Sample Size Trade-off

Once you have the basic calculation working, you'll realize that power analysis is often a negotiation. Your manager might look at "392 per group" and tell you that you can only afford to recruit 200 people. You have three levers to pull here:

• Significance Level (sig.level): If you increase this (e.g., from 0.05 to 0.10), you accept a higher risk of a False Positive, which reduces the required sample size. I generally advise against this unless the cost of a False Positive is very low.
• Power: This is the probability of detecting an effect if it actually exists. Reducing power from 0.8 to 0.7 means you're more likely to miss a real improvement (False Negative), but you'll need fewer samples.
• Effect Size (d): You can't "change" the reality of your data, but you can change your Minimum Detectable Effect (MDE). If you decide you only care about improvements of $5 or more, your d becomes 0.25, and your required sample size drops significantly.

In my experience, the most honest way to handle this is to create a small data frame of different scenarios. Don't just give your stakeholders one number; give them a menu of options showing how the sample size changes based on the power and the effect size you're hunting for.

📋 Practical Task

Sizing a Clinical Trial for Blood Pressure Reduction

You are acting as the lead statistician for a medical trial. A new medication is designed to lower systolic blood pressure. You have the following historical data:

• Average blood pressure reduction in the control group: 5 mmHg
• Expected average reduction in the treatment group: 12 mmHg
• Estimated population standard deviation: 15 mmHg

Your Task:

1. Load the pwr library.
2. Calculate the standardized effect size (Cohen's d) based on the provided means and standard deviation.
3. Determine the required sample size per group to achieve a power of 0.9 with a significance level of 0.01 (since medical trials require higher rigor).
4. Print the final sample size, rounded up to the nearest whole number using ceiling().