What causes "operation cancelled: %w" in hashicorp/nomad?

Inspect the wrapped inner error (errors.Unwrap / %w chain) — it is the actual reason fn kept failing

operation cancelled: %w

Error message

operation cancelled: %w

What it means

WithBackoffFunc runs fn repeatedly with geometric backoff until it succeeds; if the provided ctx is cancelled or times out while retries are still failing, it returns this error wrapping the last error produced by fn (or nil-wrapping if it never ran successfully). Callers like IsReady, decryptWrappedKeyTask, and waitForKey use it to poll for readiness.

Source

Thrown at helper/backoff.go:46

	if deadline > backoffLimit {
		deadline = backoffLimit
	}

	return deadline
}

// WithBackoffFunc is a helper that runs a function with geometric backoff + a
// small jitter to a maximum backoff. It returns once the context closes, with
// the error wrapping over the error from the function.
func WithBackoffFunc(ctx context.Context, minBackoff, maxBackoff time.Duration, fn func() error) error {
	var err error
	backoff := minBackoff
	t, stop := NewSafeTimer(0)
	defer stop()
	for {
		select {
		case <-ctx.Done():
			return fmt.Errorf("operation cancelled: %w", err)
		case <-t.C:
		}

		err = fn()
		if err == nil {
			return nil
		}

		if backoff < maxBackoff {
			backoff = backoff*2 + RandomStagger(minBackoff/10)
			if backoff > maxBackoff {
				backoff = maxBackoff
			}
		}

		t.Reset(backoff)
	}
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Inspect the wrapped inner error (errors.Unwrap / %w chain) — it is the actual reason fn kept failing
Increase the context deadline/timeout passed to WithBackoffFunc if the operation simply needs more time
Fix the underlying condition fn is waiting on (e.g. unseal the key, restore the dependency) so retries succeed before cancellation
If cancellation is expected (shutdown), treat this error as normal and check errors.Is against context.Canceled/DeadlineExceeded where relevant

Example fix

// before
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
// after
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)

Defensive patterns

Strategy: try-catch

Try / catch

err := helper.WithBackoffFunc(ctx, min, max, fn)
if err != nil {
	if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
		// caller cancelled; handle shutdown gracefully
	} else {
		// inspect wrapped inner error for the real failure reason
		inner := errors.Unwrap(err)
		log.Printf("polling failed until cancellation: %v", inner)
	}
}

Prevention

Always pass a context whose deadline matches how long you are willing to wait
Check whether the operation is actually retryable before wrapping it in backoff
Log the inner fn error at each attempt so the final wrapped error is explainable
Distinguish shutdown-triggered cancellation (expected) from timeout (needs a longer deadline or a fix to fn)

When it happens

Trigger: The context passed to WithBackoffFunc is cancelled or its deadline expires before fn() returns nil — e.g. a shutdown interrupts waitForKey, or a timeout elapses while decryptWrappedKeyTask keeps failing.

Common situations: Key not yet available/unsealed when a task waits for it and the parent context deadline hits; server shutdown cancelling readiness polling; decrypt failing persistently (wrong keyring) until the timeout fires, so the last fn error surfaces wrapped here.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8e0d6b1ffedd18a3. Report an issue: GitHub.