hashicorp/nomad · error

failed to retrieve jobs for idempotency check

Error message

failed to retrieve jobs for idempotency check

What it means

CheckIdempotencyToken lists child jobs of a dispatched parent job (JobsByIDPrefix) to detect a duplicate dispatch using the same idempotency token. If the prefix iterator cannot be created, the specific underlying error is swallowed and replaced with this generic message, signaling a state-store read failure during the idempotency check.

Source

Thrown at nomad/state/state_store.go:1928

	// Insert the job
	if err := txn.Insert("jobs", job); err != nil {
		return fmt.Errorf("job insert failed: %v", err)
	}
	if err := txn.Insert("index", &IndexEntry{"jobs", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return nil
}

// CheckIdempotencyToken finds all children of the parent job ID and checks to
// make sure none of them were dispatched with the idempotency token passed as
// an argument. Returns the child job found, if any.
func (s *StateStore) CheckIdempotencyToken(ns, parentID, idempotencyToken string) (*structs.Job, error) {
	iter, err := s.JobsByIDPrefix(nil, ns, parentID, SortDefault)
	if err != nil {
		return nil, errors.New("failed to retrieve jobs for idempotency check")
	}

	for {
		raw := iter.Next()
		if raw == nil {
			break
		}
		existingDispatch := raw.(*structs.Job)
		if existingDispatch.ParentID != parentID {
			continue
		}
		if existingDispatch.DispatchIdempotencyToken == idempotencyToken {
			return existingDispatch, nil
		}
	}

	return nil, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check Nomad server logs for the underlying BoltDB/state-store error and address the root cause (disk, corruption)
2. Restart the Nomad server agent; if a snapshot is corrupted, restore from a known-good Raft snapshot/backup
3. Re-run the dispatch operation once the state store is healthy

job, err := store.CheckIdempotencyToken(ns, parentID, token)
if err != nil && strings.Contains(err.Error(), "failed to retrieve jobs for idempotency check") {
    // inspect server logs for root cause; retry after state store recovers
    return nil, fmt.Errorf("state store unhealthy: %w", err)
}

When it happens

Trigger: Calling StateStore.CheckIdempotencyToken (used by the Job.Dispatch RPC) when the underlying JobsByIDPrefix wildcard/prefix iterator fails to initialize, e.g. corrupted table or internal iteration setup error in the state store.

Common situations: State store degradation (bad snapshot restore, disk issues with BoltDB); concurrent state store issues during heavy dispatch workloads; custom embedded-Nomad usage invoking the state store directly.

Related errors

• failed to lookup state snapshot: %v
• failed to retrieve jobs for idempotency check
• node lookup failed: %v
• node update failed: %v
• volume update failed: %s: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/63737367a0eae6d0. Report an issue: GitHub.