hashicorp/nomad · error

reconciliation of job summaries failed: %v

Error message

reconciliation of job summaries failed: %v

What it means

System.ReconcileJobSummaries applies a ReconcileJobSummariesRequest through Raft to rebuild job summary counters. Any error returned from the Raft apply step (loss of leader, no quorum, FSM apply failure) is wrapped in this message. It indicates the reconciliation transaction could not be committed.

Source

Thrown at nomad/system_endpoint.go:77

	authErr := s.srv.Authenticate(s.ctx, args)
	if done, err := s.srv.forward("System.ReconcileJobSummaries", args, args, reply); done {
		return err
	}
	s.srv.MeasureRPCRate("system", structs.RateMetricWrite, args)
	if authErr != nil {
		return structs.ErrPermissionDenied
	}

	// Check management level permissions
	if aclObj, err := s.srv.ResolveACL(args); err != nil {
		return err
	} else if !aclObj.IsManagement() {
		return structs.ErrPermissionDenied
	}

	_, index, err := s.srv.raftApply(structs.ReconcileJobSummariesRequestType, args)
	if err != nil {
		return fmt.Errorf("reconciliation of job summaries failed: %v", err)
	}
	reply.Index = index
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify a stable leader exists (nomad server members / leader endpoint) and retry
2. Restore quorum by restarting or replacing failed server nodes
3. Check server logs for the underlying raftApply error
4. Retry reconciliation after the cluster converges; it is idempotent

Example fix

// before
nomad system reconcile-job-summaries  // fails during leader election

// after
nomad operator api '/v1/status/leader'  # wait for a leader
nomad system reconcile-job-summaries

leader, _ := client.Status().Leader(ctx)
if leader == "" {
    return errors.New("cannot reconcile job summaries without a leader")
}

err := client.System().ReconcileJobSummaries();
var respErr *api.ResponseError
if err != nil && strings.Contains(err.Error(), "reconciliation of job summaries failed") {
    // verify leadership/quorum, retry after convergence
}

When it happens

Trigger: Calling nomad system reconcile-job-summaries (management ACL required) when raftApply returns an error, e.g. no cluster leader, lost quorum, or the FSM rejected the request.

Common situations: Running reconciliation during a leader election or after server failures; job summary inconsistency repair attempts on a degraded cluster; permission is fine (management) but the cluster cannot commit writes.

Related errors

• failed to reset heartbeat since server is not leader
• unsupported minimum common raft protocol version
• must provide peer id or address
• No cluster leader
• raft apply failed: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a3cdd9923d612711. Report an issue: GitHub.