{"record":{"id":"fcab78a70571ce36","repo":"hibiken/asynq","slug":"asynq-task-lease-expired","errorCode":null,"errorMessage":"asynq: task lease expired","messagePattern":"asynq: task lease expired","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"recoverer.go","lineNumber":82,"sourceCode":"\t\tr.recover()\n\t\ttimer := time.NewTimer(r.interval)\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-r.done:\n\t\t\t\tr.logger.Debug(\"Recoverer done\")\n\t\t\t\ttimer.Stop()\n\t\t\t\treturn\n\t\t\tcase <-timer.C:\n\t\t\t\tr.recover()\n\t\t\t\ttimer.Reset(r.interval)\n\t\t\t}\n\t\t}\n\t}()\n}\n\n// ErrLeaseExpired error indicates that the task failed because the worker working on the task\n// could not extend its lease due to missing heartbeats. The worker may have crashed or got cutoff from the network.\nvar ErrLeaseExpired = errors.New(\"asynq: task lease expired\")\n\nfunc (r *recoverer) recover() {\n\tr.recoverLeaseExpiredTasks()\n\tr.recoverStaleAggregationSets()\n}\n\nfunc (r *recoverer) recoverLeaseExpiredTasks() {\n\t// Get all tasks which have expired 30 seconds ago or earlier to accommodate certain amount of clock skew.\n\tcutoff := time.Now().Add(-30 * time.Second)\n\tmsgs, err := r.broker.ListLeaseExpired(cutoff, r.queues...)\n\tif err != nil {\n\t\tr.logger.Warnf(\"recoverer: could not list lease expired tasks: %v\", err)\n\t\treturn\n\t}\n\tfor _, msg := range msgs {\n\t\tif msg.Retried >= msg.Retry {\n\t\t\tr.archive(msg, ErrLeaseExpired)\n\t\t} else {","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/hibiken/asynq/blob/d135f1439bee74e989b7f9b41ecd542cc87f024a/recoverer.go#L64-L100","documentation":"ErrLeaseExpired indicates that a task failed because the worker processing it could not extend its lease due to missing heartbeats — typically because the worker crashed or lost network connectivity. The recoverer later finds the task with an expired lease, re-enqueues it for another worker, and the (dead or restarting) processor passes this error to handlers for the abandoned attempt.","triggerScenarios":"The processor's lease renewal goroutine stops (worker killed, process crash, GC pause, network partition); when lease.Done() fires, processor.go:245 cancels the task context and calls handleFailedMessage with ErrLeaseExpired. It also surfaces in tests via recoverLeaseExpiredTasks when tasks outlive their lease deadline.","commonSituations":"Long-running tasks that outlive the default lease because heartbeats/renewals stalled; workers killed with SIGKILL or OOM so renewal never happens; network outages between worker and Redis longer than the lease TTL; tasks repeatedly failing with this error because a worker consistently can't renew (overloaded host, clock skew).","solutions":["Ensure tasks are short or that the worker stays healthy so lease extension heartbeats keep running.","Increase lease/heartbeat robustness: reduce worker load, fix network reliability between worker and Redis, check host clock sync (NTP).","Handle the error in your ErrorHandler and make the handler idempotent, because the task will be re-enqueued and reprocessed by another worker.","If tasks are consistently too slow, use asynq's deadline/timeout options or break the task into smaller units instead of relying on one long lease."],"exampleFix":"// before\nif err := client.Enqueue(task); err != nil { return err } // handler not idempotent; expired-lease replays double-charge\n// after\nif err := client.Enqueue(task); err != nil { return err }\nfunc handleCharge(ctx context.Context, t *asynq.Task) error {\n    id := t.ResultWriter().TaskID() // or embed an idempotency key in the payload\n    if alreadyProcessed(id) { return nil } // safe replay after ErrLeaseExpired re-enqueue\n    return process(ctx, t)\n}","handlingStrategy":"try-catch","validationCode":"// ensure task runtime fits within lease expectations; monitor heartbeat health\nif strings.HasSuffix(task.Type(), \":long\") { /* split or use dedicated long-lease configuration */ }","typeGuard":"func isLeaseExpired(err error) bool { return errors.Is(err, asynq.ErrLeaseExpired) }","tryCatchPattern":"cfg := asynq.Config{ ErrorHandler: asynq.ErrorHandlerFunc(func(ctx context.Context, t *asynq.Task, err error) {\n    if errors.Is(err, asynq.ErrLeaseExpired) {\n        log.Printf(\"task abandoned by dead worker, will be re-enqueued: %s\", t.Type())\n    }\n})}","preventionTips":["Make all task handlers idempotent — expired-lease tasks are re-enqueued and reprocessed.","Monitor worker health, memory, and connectivity to Redis so lease renewals don't stall.","Avoid extremely long-running tasks; split them or rely on deadlines/timeouts.","Keep worker hosts' clocks synchronized (NTP) to avoid premature lease expiry.","Investigate repeated ErrLeaseExpired on the same task as a sign of a systematically failing worker."],"tags":["go","lease","heartbeat","worker-crash","reliability"],"backgroundTag":"lease-expired","analyzedSha":"d135f1439bee74e989b7f9b41ecd542cc87f024a","analyzedAt":"2026-09-07T19:02:34.660Z","contentChangedAt":"2026-09-07T19:02:34.660Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}