{"record":{"id":"eb5d89f77e610ca5","repo":"hashicorp/nomad","slug":"failed-to-determine-state-store-s-index-v","errorCode":null,"errorMessage":"failed to determine state store's index: %v","messagePattern":"failed to determine state store's index: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/system_endpoint.go","lineNumber":48,"sourceCode":"\tif done, err := s.srv.forward(\"System.GarbageCollect\", args, args, reply); done {\n\t\treturn err\n\t}\n\ts.srv.MeasureRPCRate(\"system\", structs.RateMetricWrite, args)\n\tif authErr != nil {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\n\t// Check management level permissions\n\tif aclObj, err := s.srv.ResolveACL(args); err != nil {\n\t\treturn err\n\t} else if !aclObj.IsManagement() {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\n\t// Get the states current index\n\tsnapshotIndex, err := s.srv.fsm.State().LatestIndex()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to determine state store's index: %v\", err)\n\t}\n\n\ts.srv.evalBroker.Enqueue(s.srv.coreJobEval(structs.CoreJobForceGC, snapshotIndex))\n\treturn nil\n}\n\n// ReconcileJobSummaries reconciles the summaries of all the jobs in the state\n// store\nfunc (s *System) ReconcileJobSummaries(args *structs.GenericRequest, reply *structs.GenericResponse) error {\n\n\tauthErr := s.srv.Authenticate(s.ctx, args)\n\tif done, err := s.srv.forward(\"System.ReconcileJobSummaries\", args, args, reply); done {\n\t\treturn err\n\t}\n\ts.srv.MeasureRPCRate(\"system\", structs.RateMetricWrite, args)\n\tif authErr != nil {\n\t\treturn structs.ErrPermissionDenied\n\t}","sourceCodeStart":30,"sourceCodeEnd":66,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/system_endpoint.go#L30-L66","documentation":"Nomad's System.GarbageCollect RPC enqueues a core force-GC evaluation, but first asks the state store for the latest Raft index. If LatestIndex() returns an error (state store unavailable or corrupted snapshot), the RPC fails with this wrapped message. It signals an internal state store failure, not a problem with the caller's request.","triggerScenarios":"Calling the system garbage-collect endpoint (nomad system gc / System.GarbageCollect) when fsm.State().LatestIndex() fails, e.g. during Raft snapshot restore, state store corruption, or an FSM that is not yet initialized.","commonSituations":"Server startup/shutdown races, failed Raft snapshot restoration, disk corruption of the state store, or querying a server whose FSM is still catching up.","solutions":["Retry the GC call after the server has fully started and joined the cluster","Check server logs for state store/Raft errors and fix the underlying FSM issue","If corruption is suspected, restore from a known-good Raft snapshot or rebuild the server","Verify disk health and free space on the Nomad data directory"],"exampleFix":"// before\nnomad system gc  // fails against a starting/degraded server\n\n// after\nnomad node status  // confirm leader and server health first\nnomad system gc    // retry once cluster is stable","handlingStrategy":"retry","validationCode":"// check cluster health first\nleader, _ := client.Status().Leader(ctx)\nif leader == \"\" { return errors.New(\"no leader; state store may be unavailable\") }","typeGuard":null,"tryCatchPattern":"err := client.System().GarbageCollect();\nif err != nil && strings.Contains(err.Error(), \"failed to determine state store's index\") {\n    // wait for server/FSM readiness, then retry with backoff\n}","preventionTips":["Only run GC against healthy servers with an established leader","Retry transient state store errors with exponential backoff","Monitor Raft/snapshot errors in server logs"],"tags":["nomad","state-store","raft","garbage-collection"],"backgroundTag":"state-store-index-unavailable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}