hibiken/asynq · error
asynq: task lease expired
Error message
asynq: task lease expired
What it means
ErrLeaseExpired indicates that a task failed because the worker processing it could not extend its lease due to missing heartbeats — typically because the worker crashed or lost network connectivity. The recoverer later finds the task with an expired lease, re-enqueues it for another worker, and the (dead or restarting) processor passes this error to handlers for the abandoned attempt.
Solutions
- Ensure tasks are short or that the worker stays healthy so lease extension heartbeats keep running.
- Increase lease/heartbeat robustness: reduce worker load, fix network reliability between worker and Redis, check host clock sync (NTP).
- Handle the error in your ErrorHandler and make the handler idempotent, because the task will be re-enqueued and reprocessed by another worker.
- If tasks are consistently too slow, use asynq's deadline/timeout options or break the task into smaller units instead of relying on one long lease.
Example fix
// before
if err := client.Enqueue(task); err != nil { return err } // handler not idempotent; expired-lease replays double-charge
// after
if err := client.Enqueue(task); err != nil { return err }
func handleCharge(ctx context.Context, t *asynq.Task) error {
id := t.ResultWriter().TaskID() // or embed an idempotency key in the payload
if alreadyProcessed(id) { return nil } // safe replay after ErrLeaseExpired re-enqueue
return process(ctx, t)
} Defensive patterns
Strategy: try-catch
Validate before calling
// ensure task runtime fits within lease expectations; monitor heartbeat health
if strings.HasSuffix(task.Type(), ":long") { /* split or use dedicated long-lease configuration */ } Type guard
func isLeaseExpired(err error) bool { return errors.Is(err, asynq.ErrLeaseExpired) } Try / catch
cfg := asynq.Config{ ErrorHandler: asynq.ErrorHandlerFunc(func(ctx context.Context, t *asynq.Task, err error) {
if errors.Is(err, asynq.ErrLeaseExpired) {
log.Printf("task abandoned by dead worker, will be re-enqueued: %s", t.Type())
}
})} Prevention
- Make all task handlers idempotent — expired-lease tasks are re-enqueued and reprocessed.
- Monitor worker health, memory, and connectivity to Redis so lease renewals don't stall.
- Avoid extremely long-running tasks; split them or rely on deadlines/timeouts.
- Keep worker hosts' clocks synchronized (NTP) to avoid premature lease expiry.
- Investigate repeated ErrLeaseExpired on the same task as a sign of a systematically failing worker.
When it happens
Trigger: The processor's lease renewal goroutine stops (worker killed, process crash, GC pause, network partition); when lease.Done() fires, processor.go:245 cancels the task context and calls handleFailedMessage with ErrLeaseExpired. It also surfaces in tests via recoverLeaseExpiredTasks when tasks outlive their lease deadline.
Common situations: Long-running tasks that outlive the default lease because heartbeats/renewals stalled; workers killed with SIGKILL or OOM so renewal never happens; network outages between worker and Redis longer than the lease TTL; tasks repeatedly failing with this error because a worker consistently can't renew (overloaded host, clock skew).
Related errors
- task id conflicts with another task
- testutil: redis is down
- skip retry for the task
- revoke task
- handler not found for task
AI-assisted analysis of hibiken/asynq@d135f1439b (2026-09-07).
Data as JSON: /api/errors/fcab78a70571ce36.
Report an issue: GitHub.
Appendix: source
Thrown at recoverer.go:82
r.recover()
timer := time.NewTimer(r.interval)
for {
select {
case <-r.done:
r.logger.Debug("Recoverer done")
timer.Stop()
return
case <-timer.C:
r.recover()
timer.Reset(r.interval)
}
}
}()
}
// ErrLeaseExpired error indicates that the task failed because the worker working on the task
// could not extend its lease due to missing heartbeats. The worker may have crashed or got cutoff from the network.
var ErrLeaseExpired = errors.New("asynq: task lease expired")
func (r *recoverer) recover() {
r.recoverLeaseExpiredTasks()
r.recoverStaleAggregationSets()
}
func (r *recoverer) recoverLeaseExpiredTasks() {
// Get all tasks which have expired 30 seconds ago or earlier to accommodate certain amount of clock skew.
cutoff := time.Now().Add(-30 * time.Second)
msgs, err := r.broker.ListLeaseExpired(cutoff, r.queues...)
if err != nil {
r.logger.Warnf("recoverer: could not list lease expired tasks: %v", err)
return
}
for _, msg := range msgs {
if msg.Retried >= msg.Retry {
r.archive(msg, ErrLeaseExpired)
} else {View on GitHub (pinned to d135f1439b)