hibiken/asynq · error

asynq: task lease expired

Error message

asynq: task lease expired

What it means

ErrLeaseExpired indicates that a task failed because the worker processing it could not extend its lease due to missing heartbeats — typically because the worker crashed or lost network connectivity. The recoverer later finds the task with an expired lease, re-enqueues it for another worker, and the (dead or restarting) processor passes this error to handlers for the abandoned attempt.

Solutions

  1. Ensure tasks are short or that the worker stays healthy so lease extension heartbeats keep running.
  2. Increase lease/heartbeat robustness: reduce worker load, fix network reliability between worker and Redis, check host clock sync (NTP).
  3. Handle the error in your ErrorHandler and make the handler idempotent, because the task will be re-enqueued and reprocessed by another worker.
  4. If tasks are consistently too slow, use asynq's deadline/timeout options or break the task into smaller units instead of relying on one long lease.

Example fix

// before
if err := client.Enqueue(task); err != nil { return err } // handler not idempotent; expired-lease replays double-charge
// after
if err := client.Enqueue(task); err != nil { return err }
func handleCharge(ctx context.Context, t *asynq.Task) error {
    id := t.ResultWriter().TaskID() // or embed an idempotency key in the payload
    if alreadyProcessed(id) { return nil } // safe replay after ErrLeaseExpired re-enqueue
    return process(ctx, t)
}
Defensive patterns

Strategy: try-catch

Validate before calling

// ensure task runtime fits within lease expectations; monitor heartbeat health
if strings.HasSuffix(task.Type(), ":long") { /* split or use dedicated long-lease configuration */ }

Type guard

func isLeaseExpired(err error) bool { return errors.Is(err, asynq.ErrLeaseExpired) }

Try / catch

cfg := asynq.Config{ ErrorHandler: asynq.ErrorHandlerFunc(func(ctx context.Context, t *asynq.Task, err error) {
    if errors.Is(err, asynq.ErrLeaseExpired) {
        log.Printf("task abandoned by dead worker, will be re-enqueued: %s", t.Type())
    }
})}

Prevention

When it happens

Trigger: The processor's lease renewal goroutine stops (worker killed, process crash, GC pause, network partition); when lease.Done() fires, processor.go:245 cancels the task context and calls handleFailedMessage with ErrLeaseExpired. It also surfaces in tests via recoverLeaseExpiredTasks when tasks outlive their lease deadline.

Common situations: Long-running tasks that outlive the default lease because heartbeats/renewals stalled; workers killed with SIGKILL or OOM so renewal never happens; network outages between worker and Redis longer than the lease TTL; tasks repeatedly failing with this error because a worker consistently can't renew (overloaded host, clock skew).

Related errors


AI-assisted analysis of hibiken/asynq@d135f1439b (2026-09-07). Data as JSON: /api/errors/fcab78a70571ce36. Report an issue: GitHub.

Appendix: source

Thrown at recoverer.go:82

		r.recover()
		timer := time.NewTimer(r.interval)
		for {
			select {
			case <-r.done:
				r.logger.Debug("Recoverer done")
				timer.Stop()
				return
			case <-timer.C:
				r.recover()
				timer.Reset(r.interval)
			}
		}
	}()
}

// ErrLeaseExpired error indicates that the task failed because the worker working on the task
// could not extend its lease due to missing heartbeats. The worker may have crashed or got cutoff from the network.
var ErrLeaseExpired = errors.New("asynq: task lease expired")

func (r *recoverer) recover() {
	r.recoverLeaseExpiredTasks()
	r.recoverStaleAggregationSets()
}

func (r *recoverer) recoverLeaseExpiredTasks() {
	// Get all tasks which have expired 30 seconds ago or earlier to accommodate certain amount of clock skew.
	cutoff := time.Now().Add(-30 * time.Second)
	msgs, err := r.broker.ListLeaseExpired(cutoff, r.queues...)
	if err != nil {
		r.logger.Warnf("recoverer: could not list lease expired tasks: %v", err)
		return
	}
	for _, msg := range msgs {
		if msg.Retried >= msg.Retry {
			r.archive(msg, ErrLeaseExpired)
		} else {

View on GitHub (pinned to d135f1439b)