A few years back, I worked with a dev who had built a perfectly fine "daily report" system using a simple Go binary and a cron tab on a single VPS. It worked great—until the company grew and we moved to a load-balanced cluster of five servers. He deployed the same binary to all of them, thinking he was adding redundancy. The result? Every single customer received the same "Daily Summary" email five times every morning. The support tickets flooded in within ten minutes. He hadn't built a distributed scheduler; he'd built a distributed spam machine.

When you move from a single machine to a cluster, you can't rely on local timers or OS-level crons. You need a way to ensure that even if you have a hundred workers listening for a task, only one of them actually executes it. This is the core challenge of distributed scheduling: maintaining a single source of truth for task state while avoiding the bottleneck of a single point of failure.

Managing Task Ownership with Atomic Claims

To prevent the "five emails" problem, you need a shared state—usually a database like PostgreSQL or a key-value store like Redis. Instead of a worker just seeing a task is "due" and running it, the worker must "claim" the task. In Go, we handle this by wrapping the claim logic in a transaction or using an atomic update.

I usually recommend a "state-machine" approach in your database table. You don't just have a status column; you have an owner_id and a locked_at timestamp. A worker doesn't just look for status = 'pending'; it looks for tasks that are either pending OR have been locked for so long that the previous owner is presumed dead.

// A simplified version of how a worker claims a task
func claimTask(db *sql.DB, workerID string) (*Task, error) {
    query := `
        UPDATE tasks 
        SET owner_id = $1, locked_at = NOW(), status = 'processing'
        WHERE id = (
            SELECT id FROM tasks 
            WHERE (status = 'pending' OR (status = 'processing' AND locked_at < NOW() - INTERVAL '5 minutes'))
            ORDER BY scheduled_at ASC 
            LIMIT 1 
            FOR UPDATE SKIP LOCKED
        ) 
        RETURNING id, payload;`
    
    var task Task
    err := db.QueryRow(query, workerID).Scan(&task.ID, &task.Payload)
    return &task, err
}

Notice the FOR UPDATE SKIP LOCKED. This is a lifesaver in Postgres. It tells the database to ignore any rows that are already locked by another transaction, allowing your Go workers to scale horizontally without fighting over the same row and causing massive lock contention.

Handling the Zombie Worker Problem

Distributed systems are messy. Your worker might claim a task, start processing a heavy PDF generation, and then the Kubernetes pod gets OOM-killed or the network blips. If you just mark the task as 'processing', that task is now a "zombie"—it will never be finished, and no other worker will ever touch it.

This is why the locked_at timestamp I mentioned earlier is non-negotiable. You implement a "lease" system. When a worker picks up a task, it owns it for a specific window (e.g., 5 minutes). If the task isn't marked as 'completed' within that window, the lease expires. Other workers, during their polling cycle, will see that the task is overdue and "steal" it back.

I've seen people try to solve this with complex heartbeat signals sent from the worker to the DB every few seconds. Don't do that unless you absolutely have to. It adds unnecessary noise to your network. A simple lease timeout is usually enough for 99% of use cases.

Structuring the Worker Loop

In Go, the most robust way to implement the worker is a for loop paired with a time.Ticker and a context.Context for graceful shutdowns. You don't want to kill a worker mid-task just because you're deploying a new version of the code.

The worker should poll the database, attempt to claim a task, and then execute it in a separate goroutine (or sequentially, depending on your throughput needs). Using a select block allows the worker to stay responsive to shutdown signals while still adhering to the polling interval.

func startWorker(ctx context.Context, db *sql.DB, id string) {
    ticker := time.NewTicker(10 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            task, err := claimTask(db, id)
            if err != nil {
                continue // No tasks available or DB error
            }
            processTask(task)
        }
    }
}

One final tip: keep your processTask logic idempotent. In a distributed system, "exactly once" delivery is a myth. You should aim for "at least once" delivery and ensure that if a task is run twice (because a lease expired just as the first worker finished), it doesn't cause side effects like double-charging a credit card.

📋 Practical Task

Exercise: Implementing a Lease-Based Task Claim Mechanism

You are provided with a mock database structure (a simple slice of structs and a mutex) that simulates a task table. Your goal is to implement the ClaimTask function to ensure that no two workers can claim the same task simultaneously, and that "zombie" tasks can be reclaimed.

Requirements:

• Implement ClaimTask(workerID string) *Task.
• A task can be claimed if its status is "pending".
• A task can be reclaimed if its status is "processing" but the LastHeartbeat was more than 30 seconds ago.
• Once claimed, the task's OwnerID must be set to the workerID, its status to "processing", and the LastHeartbeat updated to the current time.
• The function must be thread-safe to handle multiple goroutines acting as workers.

type Task struct {
    ID            int
    Status        string // "pending", "processing", "completed"
    OwnerID       string
    LastHeartbeat time.Time
}

type TaskStore struct {
    sync.Mutex
    Tasks []*Task
}

// TODO: Implement this method
func (s *TaskStore) ClaimTask(workerID string) *Task {
    // Your implementation here
}