hashicorp/nomad · error

failed to retrieve jobs for idempotency check

Error message

failed to retrieve jobs for idempotency check

What it means

When the request carries an IdempotencyToken, Dispatch consults snap.CheckIdempotencyToken to find an existing job created with the same token. If that state-store query itself fails, Dispatch logs the underlying error and returns this wrapped message.

Source

Thrown at nomad/job_endpoint.go:2053

	if args.Priority == 0 {
		args.Priority = parameterizedJob.Priority
	}

	// Validate the arguments and parameterized job
	agentConfig := j.srv.config
	if err := validateDispatchRequest(args, parameterizedJob, agentConfig); err != nil {
		return err
	}

	// Avoid creating new dispatched jobs for retried requests, by using the
	// idempotency token
	if args.IdempotencyToken != "" {
		found, err := snap.CheckIdempotencyToken(
			parameterizedJob.Namespace, parameterizedJob.ID, args.IdempotencyToken)
		if err != nil {
			const errMsg = "failed to retrieve jobs for idempotency check"
			j.logger.Error(errMsg, "error", err)
			return fmt.Errorf(errMsg)
		}
		if found != nil {
			// The existing job has not yet been garbage collected. Registering
			// a new job would violate the idempotency token. Return the ID and
			// index of the existing job.
			reply.JobCreateIndex = found.CreateIndex
			reply.DispatchedJobID = found.ID
			reply.Index = found.ModifyIndex
			return nil
		}
	}

	// Derive the child job and commit it via Raft - with initial status
	dispatchJob := parameterizedJob.Copy()
	dispatchJob.ID = structs.DispatchedID(parameterizedJob.ID, args.IdPrefixTemplate, time.Now())
	dispatchJob.ParentID = parameterizedJob.ID
	dispatchJob.Name = dispatchJob.ID
	dispatchJob.SetSubmitTime()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the Nomad server logs for the underlying error logged alongside this message
2. Retry the dispatch after server health is restored
3. If persistent, investigate FSM/state store health (raft stats, disk I/O) on the leader

if err != nil && strings.Contains(err.Error(), "failed to retrieve jobs for idempotency check") {
    // transient/state-store issue: back off and retry with the same IdempotencyToken
}

When it happens

Trigger: Dispatch with IdempotencyToken set while the state store query fails (internal state store / BoltDB errors, snapshot read issues) - not a user input problem.

Common situations: Nomad server under storage distress or during Raft/state-store issues; rarely hit; investigate server logs for the paired 'error' field.

Related errors

• failed to retrieve jobs for idempotency check
• error getting plugin: %s, %v
• failed to determine last evaluation index for job %q: %v
• failed to snapshot state: %v
• failed to find allocs for '%s': %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/2870c8b46d17f3d5. Report an issue: GitHub.