{"record":{"id":"63737367a0eae6d0","repo":"hashicorp/nomad","slug":"failed-to-retrieve-jobs-for-idempotency-check","errorCode":null,"errorMessage":"failed to retrieve jobs for idempotency check","messagePattern":"failed to retrieve jobs for idempotency check","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":1928,"sourceCode":"\n\t// Insert the job\n\tif err := txn.Insert(\"jobs\", job); err != nil {\n\t\treturn fmt.Errorf(\"job insert failed: %v\", err)\n\t}\n\tif err := txn.Insert(\"index\", &IndexEntry{\"jobs\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn nil\n}\n\n// CheckIdempotencyToken finds all children of the parent job ID and checks to\n// make sure none of them were dispatched with the idempotency token passed as\n// an argument. Returns the child job found, if any.\nfunc (s *StateStore) CheckIdempotencyToken(ns, parentID, idempotencyToken string) (*structs.Job, error) {\n\titer, err := s.JobsByIDPrefix(nil, ns, parentID, SortDefault)\n\tif err != nil {\n\t\treturn nil, errors.New(\"failed to retrieve jobs for idempotency check\")\n\t}\n\n\tfor {\n\t\traw := iter.Next()\n\t\tif raw == nil {\n\t\t\tbreak\n\t\t}\n\t\texistingDispatch := raw.(*structs.Job)\n\t\tif existingDispatch.ParentID != parentID {\n\t\t\tcontinue\n\t\t}\n\t\tif existingDispatch.DispatchIdempotencyToken == idempotencyToken {\n\t\t\treturn existingDispatch, nil\n\t\t}\n\t}\n\n\treturn nil, nil\n}","sourceCodeStart":1910,"sourceCodeEnd":1946,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L1910-L1946","documentation":"CheckIdempotencyToken lists child jobs of a dispatched parent job (JobsByIDPrefix) to detect a duplicate dispatch using the same idempotency token. If the prefix iterator cannot be created, the specific underlying error is swallowed and replaced with this generic message, signaling a state-store read failure during the idempotency check.","triggerScenarios":"Calling StateStore.CheckIdempotencyToken (used by the Job.Dispatch RPC) when the underlying JobsByIDPrefix wildcard/prefix iterator fails to initialize, e.g. corrupted table or internal iteration setup error in the state store.","commonSituations":"State store degradation (bad snapshot restore, disk issues with BoltDB); concurrent state store issues during heavy dispatch workloads; custom embedded-Nomad usage invoking the state store directly.","solutions":["Check Nomad server logs for the underlying BoltDB/state-store error and address the root cause (disk, corruption)","Restart the Nomad server agent; if a snapshot is corrupted, restore from a known-good Raft snapshot/backup","Re-run the dispatch operation once the state store is healthy"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"job, err := store.CheckIdempotencyToken(ns, parentID, token)\nif err != nil && strings.Contains(err.Error(), \"failed to retrieve jobs for idempotency check\") {\n    // inspect server logs for root cause; retry after state store recovers\n    return nil, fmt.Errorf(\"state store unhealthy: %w\", err)\n}","preventionTips":["Monitor Nomad server disk health and BoltDB errors","Restore from good snapshots after corruption","Retry dispatch operations on transient state-store failures"],"tags":["nomad","state-store","idempotency","dispatch","storage"],"backgroundTag":"state-store-read-failure","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}