hashicorp/nomad · error

failed to determine state store's index: %v

Error message

failed to determine state store's index: %v

What it means

Nomad's System.GarbageCollect RPC enqueues a core force-GC evaluation, but first asks the state store for the latest Raft index. If LatestIndex() returns an error (state store unavailable or corrupted snapshot), the RPC fails with this wrapped message. It signals an internal state store failure, not a problem with the caller's request.

Source

Thrown at nomad/system_endpoint.go:48

	if done, err := s.srv.forward("System.GarbageCollect", args, args, reply); done {
		return err
	}
	s.srv.MeasureRPCRate("system", structs.RateMetricWrite, args)
	if authErr != nil {
		return structs.ErrPermissionDenied
	}

	// Check management level permissions
	if aclObj, err := s.srv.ResolveACL(args); err != nil {
		return err
	} else if !aclObj.IsManagement() {
		return structs.ErrPermissionDenied
	}

	// Get the states current index
	snapshotIndex, err := s.srv.fsm.State().LatestIndex()
	if err != nil {
		return fmt.Errorf("failed to determine state store's index: %v", err)
	}

	s.srv.evalBroker.Enqueue(s.srv.coreJobEval(structs.CoreJobForceGC, snapshotIndex))
	return nil
}

// ReconcileJobSummaries reconciles the summaries of all the jobs in the state
// store
func (s *System) ReconcileJobSummaries(args *structs.GenericRequest, reply *structs.GenericResponse) error {

	authErr := s.srv.Authenticate(s.ctx, args)
	if done, err := s.srv.forward("System.ReconcileJobSummaries", args, args, reply); done {
		return err
	}
	s.srv.MeasureRPCRate("system", structs.RateMetricWrite, args)
	if authErr != nil {
		return structs.ErrPermissionDenied
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the GC call after the server has fully started and joined the cluster
2. Check server logs for state store/Raft errors and fix the underlying FSM issue
3. If corruption is suspected, restore from a known-good Raft snapshot or rebuild the server
4. Verify disk health and free space on the Nomad data directory

Example fix

// before
nomad system gc  // fails against a starting/degraded server

// after
nomad node status  // confirm leader and server health first
nomad system gc    // retry once cluster is stable

// check cluster health first
leader, _ := client.Status().Leader(ctx)
if leader == "" { return errors.New("no leader; state store may be unavailable") }

err := client.System().GarbageCollect();
if err != nil && strings.Contains(err.Error(), "failed to determine state store's index") {
    // wait for server/FSM readiness, then retry with backoff
}

When it happens

Trigger: Calling the system garbage-collect endpoint (nomad system gc / System.GarbageCollect) when fsm.State().LatestIndex() fails, e.g. during Raft snapshot restore, state store corruption, or an FSM that is not yet initialized.

Common situations: Server startup/shutdown races, failed Raft snapshot restoration, disk corruption of the state store, or querying a server whose FSM is still catching up.

Related errors

• error querying plugin %q: %v
• deployment id not found: %q
• index update failed: %v
• JobByID lookup failed: %w
• UpsertJob failed: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/eb5d89f77e610ca5. Report an issue: GitHub.