hashicorp/nomad · critical

recovery failed: %v

Error message

recovery failed: %v

What it means

This is the top-level failure of raft.RecoverCluster: after peers.json parsed and a temp FSM was created, actually rewriting the Raft log/snapshot metadata with the new cluster configuration failed. RecoverCluster can fail on inconsistent logs/snapshots, mismatched store formats, or an invalid configuration object.

Source

Thrown at nomad/server.go:1565

			}
		} else if _, err := os.Stat(peersFile); err == nil {
			s.logger.Info("found peers.json file, recovering Raft configuration...")
			var configuration raft.Configuration
			if s.config.RaftConfig.ProtocolVersion < 3 {
				configuration, err = raft.ReadPeersJSON(peersFile)
			} else {
				configuration, err = raft.ReadConfigJSON(peersFile)
			}
			if err != nil {
				return fmt.Errorf("recovery failed to parse peers.json: %v", err)
			}
			tmpFsm, err := NewFSM(fsmConfig)
			if err != nil {
				return fmt.Errorf("recovery failed to make temp FSM: %v", err)
			}
			if err := raft.RecoverCluster(s.config.RaftConfig, tmpFsm,
				log, stable, snap, trans, configuration); err != nil {
				return fmt.Errorf("recovery failed: %v", err)
			}
			if err := os.Remove(peersFile); err != nil {
				return fmt.Errorf("recovery failed to delete peers.json, please delete manually (see peers.info for details): %v", err)
			}
			s.logger.Info("deleted peers.json file after successful recovery")
		}
	}

	// If we are a single server cluster and the state is clean then we can
	// bootstrap now.
	if s.isSingleServerCluster() {
		hasState, err := raft.HasExistingState(log, stable, snap)
		if err != nil {
			return err
		}
		if !hasState {
			configuration := raft.Configuration{
				Servers: []raft.Server{

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Confirm every server ID and address in peers.json matches real servers and that their Raft protocol versions match the config (protocol_version = 3 for modern Nomad)
2. Run recovery on a verified backup copy of the data dir; retry with the latest Nomad version
3. Check that the raft/ dir contains both a valid snapshot and log store; restore from a known-good backup if not
4. As a last resort, wipe the data dir and rebootstrap (losing state) after taking the peers.json recovery out of play

Example fix

// before (mismatched protocol)
# peers.json lists servers, but server config: protocol_version = 2
// after
# set in all server stanzas
protocol_version = 3
sudo systemctl restart nomad

// Pre-flight: protocol version in config must match stored data, peers consistent
const cfgVersion = 3; // protocol_version from server config
const peerIds = JSON.parse(fs.readFileSync(dataDir + '/peers.json','utf8')).Servers.map(s => s.ID);
if (new Set(peerIds).size !== peerIds.length) throw new Error('duplicate server IDs in peers.json');
if (cfgVersion !== 3) throw new Error('RecoverCluster requires consistent raft protocol (use 3)');

try {
  startNomadServer();
} catch (e) {
  if (String(e).includes('recovery failed:')) {
    // recover on a fresh backup copy with verified peers.json
    restoreDataDirFromBackup();
    rewritePeersJsonFromVerifiedMembers();
    startNomadServer();
  } else throw e;
}

When it happens

Trigger: Server start with peers.json present (no peers.info) and raft.RecoverCluster returns an error — snapshot/log corruption, Raft protocol version mismatch between config and stored data, or an inconsistent peers.json vs actual server IDs.

Common situations: Attempting recovery after restoring snapshots from a different cluster; mixed Nomad versions where Raft protocol versions differ among peers listed in peers.json; partially failed upgrade left inconsistent raft state; recovery attempted against a data dir already recovered once.

Related errors

• No cluster leader
• failed to read log entry at index %d (firstIdx: %d, lastIdx:
• command did not include data
• failed to delete peers.json, please delete manually (see pee
• recovery failed to parse peers.json: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0e2d56c859856818. Report an issue: GitHub.