{"record":{"id":"930277cded72dd63","repo":"hashicorp/nomad","slug":"recovery-failed-to-delete-peers-json-please-delet","errorCode":null,"errorMessage":"recovery failed to delete peers.json, please delete manually (see peers.info for details): %v","messagePattern":"recovery failed to delete peers\\.json, please delete manually \\(see peers\\.info for details\\): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/server.go","lineNumber":1568,"sourceCode":"\t\t\tvar configuration raft.Configuration\n\t\t\tif s.config.RaftConfig.ProtocolVersion < 3 {\n\t\t\t\tconfiguration, err = raft.ReadPeersJSON(peersFile)\n\t\t\t} else {\n\t\t\t\tconfiguration, err = raft.ReadConfigJSON(peersFile)\n\t\t\t}\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed to parse peers.json: %v\", err)\n\t\t\t}\n\t\t\ttmpFsm, err := NewFSM(fsmConfig)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed to make temp FSM: %v\", err)\n\t\t\t}\n\t\t\tif err := raft.RecoverCluster(s.config.RaftConfig, tmpFsm,\n\t\t\t\tlog, stable, snap, trans, configuration); err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed: %v\", err)\n\t\t\t}\n\t\t\tif err := os.Remove(peersFile); err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed to delete peers.json, please delete manually (see peers.info for details): %v\", err)\n\t\t\t}\n\t\t\ts.logger.Info(\"deleted peers.json file after successful recovery\")\n\t\t}\n\t}\n\n\t// If we are a single server cluster and the state is clean then we can\n\t// bootstrap now.\n\tif s.isSingleServerCluster() {\n\t\thasState, err := raft.HasExistingState(log, stable, snap)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tif !hasState {\n\t\t\tconfiguration := raft.Configuration{\n\t\t\t\tServers: []raft.Server{\n\t\t\t\t\t{\n\t\t\t\t\t\tID:      s.config.RaftConfig.LocalID,\n\t\t\t\t\t\tAddress: trans.LocalAddr(),","sourceCodeStart":1550,"sourceCodeEnd":1586,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/server.go#L1550-L1586","documentation":"After raft.RecoverCluster succeeds, Nomad deletes the now-consumed peers.json. If os.Remove fails at this point the recovery itself worked, but the stale file remains and would trigger another recovery pass on next start, so startup aborts and asks the operator to remove it manually.","triggerScenarios":"Immediately after successful RecoverCluster, os.Remove(peersFile) errors — file deleted out from under the process, read-only filesystem, or permission change during startup.","commonSituations":"Filesystem flipped to read-only mid-startup (e.g. LVM/RAID errors); another process or cleanup job removed/chmod'ed the file; data dir mounted with restrictive ACLs on shared storage.","solutions":["Delete peers.json manually (path is in the log line) and restart Nomad — recovery already succeeded","Check the mount is not read-only (mount | grep <data_dir>) and remount read-write","Fix ownership/ACLs on the data dir for the Nomad service user","Remove any cleanup automation touching peers.json while Nomad starts"],"exampleFix":"// before\njournalctl -u nomad  # recovery failed to delete peers.json ...\n// after\nsudo rm /var/nomad/data/peers.json\nsudo systemctl restart nomad","handlingStrategy":"validation","validationCode":"// Ensure the recovery file is deletable before kicking off recovery\nconst f = dataDir + '/peers.json';\nif (fs.existsSync(f)) fs.accessSync(f, fs.constants.W_OK); // throws if not writable/deletable","typeGuard":null,"tryCatchPattern":"try {\n  startNomadServer();\n} catch (e) {\n  if (String(e).includes('recovery failed to delete peers.json')) {\n    fs.rmSync(dataDir + '/peers.json', { force: true }); // safe: recovery already succeeded\n    startNomadServer();\n  } else throw e;\n}","preventionTips":["Don't run file cleanup jobs against the Nomad data dir while agents start","Verify the mount isn't flipped read-only (mount options, RAID health)","Keep data dir ownership consistent with the Nomad service user"],"tags":["raft","recovery","filesystem","permissions"],"backgroundTag":"raft-peers-json-recovery","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}