When people first try to implement a rate limiter in Java, they almost always start with a synchronized method and a simple integer counter. They think, "I'll just increment a counter every time a request comes in, and if it hits 100 within a second, I'll return a 429 Too Many Requests." It sounds logical, but in a high-concurrency environment, this is a recipe for a performance nightmare.

The "Synchronized Counter" Bottleneck

The problem is lock contention. If you have 50 threads all hitting a synchronized block just to check if they are allowed to proceed, you've effectively turned your multi-threaded application into a single-threaded one at the most critical entry point. You aren't just limiting the rate of requests; you're limiting the throughput of your entire CPU because every thread is fighting for that one monitor lock.

// Don't do this in a high-traffic system
public synchronized boolean allowRequest() {
    if (requestCount < MAX_REQUESTS) {
        requestCount++;
        return true;
    }
    return false;
}


I've seen this cause production outages where the rate limiter itself became the primary source of latency. The threads spend more time waiting for the lock than they do actually processing the business logic.

Using Semaphores and Scheduled Refills
To do this properly, we need to decouple the checking of the limit from the resetting of the limit. This is where the "Token Bucket" algorithm comes in. Imagine a bucket that holds tokens. Every request consumes a token. If the bucket is empty, the request is rejected. A separate process adds tokens back into the bucket at a fixed rate.

In Java, a Semaphore is a fantastic tool for this. It maintains a set of permits. When you call tryAcquire(), it returns immediately with true if a permit was available, or false if it wasn't. There's no heavy synchronization block wrapping the whole logic; the Semaphore handles the concurrency internally using highly optimized CAS (Compare-And-Swap) operations.

But a Semaphore doesn't refill itself. That's where the ScheduledExecutorService comes in. By scheduling a task to run every second (or millisecond), you can "top up" the permits in the bucket.

Here is the mental model you should use:

• The Semaphore: Acts as your bucket of tokens. It allows threads to "grab and go" without blocking each other excessively.
• The ScheduledExecutorService: Acts as the refill mechanism, ensuring the bucket is replenished regardless of how many requests are hitting the system.
• tryAcquire(): This is your gatekeeper. It's non-blocking, meaning your API remains responsive even when the rate limit is hit.

One little tip: be careful with how you refill. If you simply call release(MAX_PERMITS) every second, you might accidentally grow the semaphore beyond its original capacity if some permits weren't used. You'll want to track how many permits were actually consumed and only release that specific amount, or use a different atomic counter to track the current "fill level."

📋 Practical Task

Implement a Token Bucket Rate Limiter with a Burst Capacity

Your task is to build a TokenBucketLimiter class that manages API request limits. Your implementation must support "burstiness"—meaning the bucket can hold more tokens than are added per second, allowing a sudden spike of traffic to pass through before the steady-state limit kicks in.

Requirements:

• The class should take maxBurstSize (maximum tokens the bucket can hold) and refillRatePerSecond (how many tokens are added every second) in its constructor.
• Implement a method boolean allowRequest() that returns true if a token is available and consumes it, otherwise returns false.
• Use a ScheduledExecutorService to refill the permits every second.
• Ensure the bucket never exceeds the maxBurstSize during a refill.
• The allowRequest method must be non-blocking (do not use acquire(), use tryAcquire()).

Test Scenario: Create a main method that simulates 20 concurrent threads hitting the limiter. Set the refill rate to 5 per second and the burst size to 10. You should see the first 10 requests pass immediately, and subsequent requests pass only as the scheduler refills the bucket.