hashicorp/nomad · critical

recovery failed

Error message

recovery failed: %v

What it means

This is the top-level failure of raft.RecoverCluster: after peers.json parsed and a temp FSM was created, actually rewriting the Raft log/snapshot metadata with the new cluster configuration failed. RecoverCluster can fail on inconsistent logs/snapshots, mismatched store formats, or an invalid configuration object.

Solutions

1. Confirm every server ID and address in peers.json matches real servers and that their Raft protocol versions match the config (protocol_version = 3 for modern Nomad)
2. Run recovery on a verified backup copy of the data dir; retry with the latest Nomad version
3. Check that the raft/ dir contains both a valid snapshot and log store; restore from a known-good backup if not
4. As a last resort, wipe the data dir and rebootstrap (losing state) after taking the peers.json recovery out of play

Example fix

// before (mismatched protocol)
# peers.json lists servers, but server config: protocol_version = 2
// after
# set in all server stanzas
protocol_version = 3
sudo systemctl restart nomad

// Pre-flight: protocol version in config must match stored data, peers consistent
const cfgVersion = 3; // protocol_version from server config
const peerIds = JSON.parse(fs.readFileSync(dataDir + '/peers.json','utf8')).Servers.map(s => s.ID);
if (new Set(peerIds).size !== peerIds.length) throw new Error('duplicate server IDs in peers.json');
if (cfgVersion !== 3) throw new Error('RecoverCluster requires consistent raft protocol (use 3)');

try {
  startNomadServer();
} catch (e) {
  if (String(e).includes('recovery failed:')) {
    // recover on a fresh backup copy with verified peers.json
    restoreDataDirFromBackup();
    rewritePeersJsonFromVerifiedMembers();
    startNomadServer();
  } else throw e;
}

When it happens

Trigger: Server start with peers.json present (no peers.info) and raft.RecoverCluster returns an error — snapshot/log corruption, Raft protocol version mismatch between config and stored data, or an inconsistent peers.json vs actual server IDs.

Common situations: Attempting recovery after restoring snapshots from a different cluster; mixed Nomad versions where Raft protocol versions differ among peers listed in peers.json; partially failed upgrade left inconsistent raft state; recovery attempted against a data dir already recovered once.

Related errors

• command did not include data
• failed to delete peers.json, please delete manually (see…
• failed to read log entry at index
• No cluster leader
• recovery failed to delete peers.json, please delete…

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0e2d56c859856818. Report an issue: GitHub.

Appendix: source

Thrown at nomad/server.go:1565

			}
		} else if _, err := os.Stat(peersFile); err == nil {
			s.logger.Info("found peers.json file, recovering Raft configuration...")
			var configuration raft.Configuration
			if s.config.RaftConfig.ProtocolVersion < 3 {
				configuration, err = raft.ReadPeersJSON(peersFile)
			} else {
				configuration, err = raft.ReadConfigJSON(peersFile)
			}
			if err != nil {
				return fmt.Errorf("recovery failed to parse peers.json: %v", err)
			}
			tmpFsm, err := NewFSM(fsmConfig)
			if err != nil {
				return fmt.Errorf("recovery failed to make temp FSM: %v", err)
			}
			if err := raft.RecoverCluster(s.config.RaftConfig, tmpFsm,
				log, stable, snap, trans, configuration); err != nil {
				return fmt.Errorf("recovery failed: %v", err)
			}
			if err := os.Remove(peersFile); err != nil {
				return fmt.Errorf("recovery failed to delete peers.json, please delete manually (see peers.info for details): %v", err)
			}
			s.logger.Info("deleted peers.json file after successful recovery")
		}
	}

	// If we are a single server cluster and the state is clean then we can
	// bootstrap now.
	if s.isSingleServerCluster() {
		hasState, err := raft.HasExistingState(log, stable, snap)
		if err != nil {
			return err
		}
		if !hasState {
			configuration := raft.Configuration{
				Servers: []raft.Server{

View on GitHub (pinned to 482b49bf1a)