{"record":{"id":"8e0d6b1ffedd18a3","repo":"hashicorp/nomad","slug":"operation-cancelled-w","errorCode":null,"errorMessage":"operation cancelled: %w","messagePattern":"operation cancelled: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"helper/backoff.go","lineNumber":46,"sourceCode":"\tif deadline > backoffLimit {\n\t\tdeadline = backoffLimit\n\t}\n\n\treturn deadline\n}\n\n// WithBackoffFunc is a helper that runs a function with geometric backoff + a\n// small jitter to a maximum backoff. It returns once the context closes, with\n// the error wrapping over the error from the function.\nfunc WithBackoffFunc(ctx context.Context, minBackoff, maxBackoff time.Duration, fn func() error) error {\n\tvar err error\n\tbackoff := minBackoff\n\tt, stop := NewSafeTimer(0)\n\tdefer stop()\n\tfor {\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn fmt.Errorf(\"operation cancelled: %w\", err)\n\t\tcase <-t.C:\n\t\t}\n\n\t\terr = fn()\n\t\tif err == nil {\n\t\t\treturn nil\n\t\t}\n\n\t\tif backoff < maxBackoff {\n\t\t\tbackoff = backoff*2 + RandomStagger(minBackoff/10)\n\t\t\tif backoff > maxBackoff {\n\t\t\t\tbackoff = maxBackoff\n\t\t\t}\n\t\t}\n\n\t\tt.Reset(backoff)\n\t}\n}","sourceCodeStart":28,"sourceCodeEnd":64,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/backoff.go#L28-L64","documentation":"WithBackoffFunc runs fn repeatedly with geometric backoff until it succeeds; if the provided ctx is cancelled or times out while retries are still failing, it returns this error wrapping the last error produced by fn (or nil-wrapping if it never ran successfully). Callers like IsReady, decryptWrappedKeyTask, and waitForKey use it to poll for readiness.","triggerScenarios":"The context passed to WithBackoffFunc is cancelled or its deadline expires before fn() returns nil — e.g. a shutdown interrupts waitForKey, or a timeout elapses while decryptWrappedKeyTask keeps failing.","commonSituations":"Key not yet available/unsealed when a task waits for it and the parent context deadline hits; server shutdown cancelling readiness polling; decrypt failing persistently (wrong keyring) until the timeout fires, so the last fn error surfaces wrapped here.","solutions":["Inspect the wrapped inner error (errors.Unwrap / %w chain) — it is the actual reason fn kept failing","Increase the context deadline/timeout passed to WithBackoffFunc if the operation simply needs more time","Fix the underlying condition fn is waiting on (e.g. unseal the key, restore the dependency) so retries succeed before cancellation","If cancellation is expected (shutdown), treat this error as normal and check errors.Is against context.Canceled/DeadlineExceeded where relevant"],"exampleFix":"// before\nctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)\n// after\nctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := helper.WithBackoffFunc(ctx, min, max, fn)\nif err != nil {\n\tif errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {\n\t\t// caller cancelled; handle shutdown gracefully\n\t} else {\n\t\t// inspect wrapped inner error for the real failure reason\n\t\tinner := errors.Unwrap(err)\n\t\tlog.Printf(\"polling failed until cancellation: %v\", inner)\n\t}\n}","preventionTips":["Always pass a context whose deadline matches how long you are willing to wait","Check whether the operation is actually retryable before wrapping it in backoff","Log the inner fn error at each attempt so the final wrapped error is explainable","Distinguish shutdown-triggered cancellation (expected) from timeout (needs a longer deadline or a fix to fn)"],"tags":["context","retry","backoff","timeout"],"backgroundTag":"context-cancelled","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}