{"record":{"id":"2870c8b46d17f3d5","repo":"hashicorp/nomad","slug":"failed-to-retrieve-jobs-for-idempotency-check-2870c8","errorCode":null,"errorMessage":"failed to retrieve jobs for idempotency check","messagePattern":"failed to retrieve jobs for idempotency check","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/job_endpoint.go","lineNumber":2053,"sourceCode":"\tif args.Priority == 0 {\n\t\targs.Priority = parameterizedJob.Priority\n\t}\n\n\t// Validate the arguments and parameterized job\n\tagentConfig := j.srv.config\n\tif err := validateDispatchRequest(args, parameterizedJob, agentConfig); err != nil {\n\t\treturn err\n\t}\n\n\t// Avoid creating new dispatched jobs for retried requests, by using the\n\t// idempotency token\n\tif args.IdempotencyToken != \"\" {\n\t\tfound, err := snap.CheckIdempotencyToken(\n\t\t\tparameterizedJob.Namespace, parameterizedJob.ID, args.IdempotencyToken)\n\t\tif err != nil {\n\t\t\tconst errMsg = \"failed to retrieve jobs for idempotency check\"\n\t\t\tj.logger.Error(errMsg, \"error\", err)\n\t\t\treturn fmt.Errorf(errMsg)\n\t\t}\n\t\tif found != nil {\n\t\t\t// The existing job has not yet been garbage collected. Registering\n\t\t\t// a new job would violate the idempotency token. Return the ID and\n\t\t\t// index of the existing job.\n\t\t\treply.JobCreateIndex = found.CreateIndex\n\t\t\treply.DispatchedJobID = found.ID\n\t\t\treply.Index = found.ModifyIndex\n\t\t\treturn nil\n\t\t}\n\t}\n\n\t// Derive the child job and commit it via Raft - with initial status\n\tdispatchJob := parameterizedJob.Copy()\n\tdispatchJob.ID = structs.DispatchedID(parameterizedJob.ID, args.IdPrefixTemplate, time.Now())\n\tdispatchJob.ParentID = parameterizedJob.ID\n\tdispatchJob.Name = dispatchJob.ID\n\tdispatchJob.SetSubmitTime()","sourceCodeStart":2035,"sourceCodeEnd":2071,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/job_endpoint.go#L2035-L2071","documentation":"When the request carries an IdempotencyToken, Dispatch consults snap.CheckIdempotencyToken to find an existing job created with the same token. If that state-store query itself fails, Dispatch logs the underlying error and returns this wrapped message.","triggerScenarios":"Dispatch with IdempotencyToken set while the state store query fails (internal state store / BoltDB errors, snapshot read issues) - not a user input problem.","commonSituations":"Nomad server under storage distress or during Raft/state-store issues; rarely hit; investigate server logs for the paired 'error' field.","solutions":["Check the Nomad server logs for the underlying error logged alongside this message","Retry the dispatch after server health is restored","If persistent, investigate FSM/state store health (raft stats, disk I/O) on the leader"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"failed to retrieve jobs for idempotency check\") {\n    // transient/state-store issue: back off and retry with the same IdempotencyToken\n}","preventionTips":["Keep IdempotencyToken stable across retries of the same logical dispatch","Watch Nomad server logs and health when using idempotent dispatch","Retry with exponential backoff; do not change the token between retries"],"tags":["nomad","api","state-store","idempotency","internal"],"backgroundTag":"state-store-read-failure","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}