hashicorp/nomad · error
failed to retrieve jobs: %w
Error message
failed to retrieve jobs: %w
What it means
UpgradeCheckVaultWorkloadIdentity scans the state store for jobs using Vault without a Vault workload identity. If state.StateStore.Jobs() fails to return the jobs iterator, the error is wrapped with %w and returned to the RPC caller.
Source
Thrown at nomad/operator_endpoint.go:774
op.srv.MeasureRPCRate("operator", structs.RateMetricRead, args)
if authErr != nil {
return structs.ErrPermissionDenied
}
// This action requires operator read access.
aclObj, err := op.srv.ResolveACL(args)
if err != nil {
return err
} else if !aclObj.AllowOperatorRead() {
return structs.ErrPermissionDenied
}
ws := memdb.NewWatchSet()
// Check for jobs that use Vault but don't have an identity for Vault.
jobsIter, err := op.srv.State().Jobs(ws, state.SortDefault)
if err != nil {
return fmt.Errorf("failed to retrieve jobs: %w", err)
}
jobs := []*structs.JobListStub{}
for raw := jobsIter.Next(); raw != nil; raw = jobsIter.Next() {
job := raw.(*structs.Job)
TG_LOOP:
for _, tg := range job.TaskGroups {
for _, t := range tg.Tasks {
if t.Vault == nil {
continue
}
foundWID := false
for _, wid := range t.Identities {
if wid.IsVault() {
foundWID = true
breakView on GitHub (pinned to 482b49bf1a)
Solutions
- Retry the operator upgrade check command
- Check server logs for the wrapped state store error
- Verify state store integrity (latest index consistency, raft state)
- Restart the server / restore from a known-good snapshot if corruption is confirmed
Defensive patterns
Strategy: retry
Try / catch
resp, err := client.Operator().UpgradeCheck(nil)
if err != nil && strings.Contains(err.Error(), "failed to retrieve jobs") {
// transient state store failure: back off and retry
} Prevention
- Avoid running upgrade checks mid-restore
- Monitor state store health on servers
- Retry with backoff before escalating
When it happens
Trigger: op.srv.State().Jobs(ws, state.SortDefault) returns a memdb/state-store error — typically an internal state store failure or invalid store snapshot state during restore/upgrade checks.
Common situations: State store corruption or a restore in progress; running the upgrade check immediately after a snapshot restore while the state DB is inconsistent.
Related errors
- failed to retrieve nodes: %w
- no signed workload identity available
- failed to retrieve signed workload identity: %w
- incompatible state version. expected %q but found %q
- error deleting unexpected key %q: %v
AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04).
Data as JSON: /api/errors/516847e4ca298be9.
Report an issue: GitHub.