{"record":{"id":"0e2d56c859856818","repo":"hashicorp/nomad","slug":"recovery-failed-v","errorCode":null,"errorMessage":"recovery failed: %v","messagePattern":"recovery failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"nomad/server.go","lineNumber":1565,"sourceCode":"\t\t\t}\n\t\t} else if _, err := os.Stat(peersFile); err == nil {\n\t\t\ts.logger.Info(\"found peers.json file, recovering Raft configuration...\")\n\t\t\tvar configuration raft.Configuration\n\t\t\tif s.config.RaftConfig.ProtocolVersion < 3 {\n\t\t\t\tconfiguration, err = raft.ReadPeersJSON(peersFile)\n\t\t\t} else {\n\t\t\t\tconfiguration, err = raft.ReadConfigJSON(peersFile)\n\t\t\t}\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed to parse peers.json: %v\", err)\n\t\t\t}\n\t\t\ttmpFsm, err := NewFSM(fsmConfig)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed to make temp FSM: %v\", err)\n\t\t\t}\n\t\t\tif err := raft.RecoverCluster(s.config.RaftConfig, tmpFsm,\n\t\t\t\tlog, stable, snap, trans, configuration); err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed: %v\", err)\n\t\t\t}\n\t\t\tif err := os.Remove(peersFile); err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed to delete peers.json, please delete manually (see peers.info for details): %v\", err)\n\t\t\t}\n\t\t\ts.logger.Info(\"deleted peers.json file after successful recovery\")\n\t\t}\n\t}\n\n\t// If we are a single server cluster and the state is clean then we can\n\t// bootstrap now.\n\tif s.isSingleServerCluster() {\n\t\thasState, err := raft.HasExistingState(log, stable, snap)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tif !hasState {\n\t\t\tconfiguration := raft.Configuration{\n\t\t\t\tServers: []raft.Server{","sourceCodeStart":1547,"sourceCodeEnd":1583,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/server.go#L1547-L1583","documentation":"This is the top-level failure of raft.RecoverCluster: after peers.json parsed and a temp FSM was created, actually rewriting the Raft log/snapshot metadata with the new cluster configuration failed. RecoverCluster can fail on inconsistent logs/snapshots, mismatched store formats, or an invalid configuration object.","triggerScenarios":"Server start with peers.json present (no peers.info) and raft.RecoverCluster returns an error — snapshot/log corruption, Raft protocol version mismatch between config and stored data, or an inconsistent peers.json vs actual server IDs.","commonSituations":"Attempting recovery after restoring snapshots from a different cluster; mixed Nomad versions where Raft protocol versions differ among peers listed in peers.json; partially failed upgrade left inconsistent raft state; recovery attempted against a data dir already recovered once.","solutions":["Confirm every server ID and address in peers.json matches real servers and that their Raft protocol versions match the config (protocol_version = 3 for modern Nomad)","Run recovery on a verified backup copy of the data dir; retry with the latest Nomad version","Check that the raft/ dir contains both a valid snapshot and log store; restore from a known-good backup if not","As a last resort, wipe the data dir and rebootstrap (losing state) after taking the peers.json recovery out of play"],"exampleFix":"// before (mismatched protocol)\n# peers.json lists servers, but server config: protocol_version = 2\n// after\n# set in all server stanzas\nprotocol_version = 3\nsudo systemctl restart nomad","handlingStrategy":"validation","validationCode":"// Pre-flight: protocol version in config must match stored data, peers consistent\nconst cfgVersion = 3; // protocol_version from server config\nconst peerIds = JSON.parse(fs.readFileSync(dataDir + '/peers.json','utf8')).Servers.map(s => s.ID);\nif (new Set(peerIds).size !== peerIds.length) throw new Error('duplicate server IDs in peers.json');\nif (cfgVersion !== 3) throw new Error('RecoverCluster requires consistent raft protocol (use 3)');","typeGuard":null,"tryCatchPattern":"try {\n  startNomadServer();\n} catch (e) {\n  if (String(e).includes('recovery failed:')) {\n    // recover on a fresh backup copy with verified peers.json\n    restoreDataDirFromBackup();\n    rewritePeersJsonFromVerifiedMembers();\n    startNomadServer();\n  } else throw e;\n}","preventionTips":["Take a full data-dir backup before any recovery attempt","Ensure all servers in peers.json share the same Raft protocol version as the config","Never recover a data dir that was already recovered once or mixes snapshots from different clusters","Upgrade to the latest Nomad before attempting recovery"],"tags":["raft","recovery","cluster","data-loss"],"backgroundTag":"raft-peers-json-recovery","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}