Imagine you've just walked into a casino and you're staring at a row of five different slot machines. You know that each machine has a different probability of paying out, but the casino isn't telling you which is which. You have 100 tokens. Now you're faced with a classic dilemma: do you spend your tokens testing every single machine to find the one with the highest payout (exploration), or do you find one that pays out once and just keep pulling that lever for the rest of the night (exploitation)?

In software engineering and data science, we call this the Multi-Armed Bandit problem. It's the core logic behind everything from A/B testing website headers to how Netflix decides which movie thumbnails to show you. If you exploit too early, you might miss the "jackpot" machine. If you explore too long, you waste tokens on losers.

Defining the 'Arms' of our Bandit

To simulate this in R, we first need to define our environment. In our case, the "arms" are just a vector of probabilities. I like to think of this as the "ground truth" that our simulation is trying to discover.

# The ground truth: 5 machines with different win probabilities
true_probs <- c(0.05, 0.12, 0.45, 0.20, 0.08)
n_arms <- length(true_probs)
n_trials <- 1000
epsilon <- 0.1 # 10% of the time we explore

Here, the third machine is the clear winner at 45%. Our code doesn't know that, though; it only knows what happens after it "pulls the lever."

The Tug-of-War: Exploration vs. Exploitation

The most straightforward way to handle this is the $\epsilon$-greedy (epsilon-greedy) algorithm. The logic is simple: most of the time, we pick the best machine we've found so far. But occasionally—controlled by $\epsilon$—we intentionally pick a random machine just to see if we were wrong about our current favorite.

Let's build the simulation loop. I'll use a vector to track our estimated means for each arm and another to track how many times we've pulled each lever.

# Initialize our knowledge
est_means <- rep(0, n_arms)
counts <- rep(0, n_arms)
total_reward <- 0

for (i in 1:n_trials) {
  # Decide: Explore or Exploit?
  if (runif(1) < epsilon) {
    # Exploration: Pick a random arm
    chosen_arm <- sample(1:n_arms, 1)
  } else {
    # Exploitation: Pick the arm with the current best estimate
    chosen_arm <- which.max(est_means)
  }
  
  # Simulate the payout (1 for win, 0 for loss)
  reward <- rbinom(1, 1, true_probs[chosen_arm])
  
  # Update our knowledge using the incremental average formula
  counts[chosen_arm] <- counts[chosen_arm] + 1
  est_means[chosen_arm] <- est_means[chosen_arm] + (reward - est_means[chosen_arm]) / counts[chosen_arm]
  
  total_reward <- total_reward + reward
}

print(paste("Total Reward:", total_reward))
print("Estimated Probabilities:")
print(est_means)

Why the Incremental Average Matters

You'll notice I didn't just sum the rewards and divide by the count inside the loop. I used est_means[chosen_arm] + (reward - est_means[chosen_arm]) / counts[chosen_arm]. This is a little trick to avoid storing a massive list of every single result. It updates the average on the fly, which is much more memory-efficient when you're running millions of trials—something you'll frequently do in production simulations.

If you run this code, you'll see that est_means usually gets very close to true_probs. But more importantly, if you look at the counts vector, you'll see the simulation spent the vast majority of its time pulling the arm with the 0.45 probability. That's the $\epsilon$-greedy strategy working: it explored enough to find the winner, then exploited that winner to maximize the total reward.

📋 Practical Task

Simulating Ad Campaign Optimization

You are managing an online ad campaign with three different banner designs (Arms A, B, and C). Your goal is to maximize the number of clicks. The actual click-through rates (CTR) are: Arm A = 2%, Arm B = 8%, and Arm C = 5%.

Write an R script to simulate 2,000 impressions using an $\epsilon$-greedy strategy. However, I want you to test the impact of the exploration rate. Run the simulation twice: once with epsilon = 0.01 (1% exploration) and once with epsilon = 0.2 (20% exploration).

Your code should output the total rewards (clicks) and the final estimated probabilities for both scenarios. In your results, observe which epsilon value led to a higher total reward and why the "over-exploring" agent might have a more accurate estimate of the losing arms but a lower total payout.