{"record":{"id":"3af4fec5f03b0ddc","repo":"hashicorp/nomad","slug":"failed-to-delete-peers-json-please-delete-manuall","errorCode":null,"errorMessage":"failed to delete peers.json, please delete manually (see peers.info for details): %v","messagePattern":"failed to delete peers\\.json, please delete manually \\(see peers\\.info for details\\): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/server.go","lineNumber":1544,"sourceCode":"\t\t// For an existing cluster being upgraded to the new version of\n\t\t// Raft, we almost never want to run recovery based on the old\n\t\t// peers.json file. We create a peers.info file with a helpful\n\t\t// note about where peers.json went, and use that as a sentinel\n\t\t// to avoid ingesting the old one that first time (if we have to\n\t\t// create the peers.info file because it's not there, we also\n\t\t// blow away any existing peers.json file).\n\t\tpeersFile := filepath.Join(path, \"peers.json\")\n\t\tpeersInfoFile := filepath.Join(path, \"peers.info\")\n\t\tif _, err := os.Stat(peersInfoFile); os.IsNotExist(err) {\n\t\t\tif err := os.WriteFile(peersInfoFile, []byte(peersInfoContent), 0644); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to write peers.info file: %v\", err)\n\t\t\t}\n\n\t\t\t// Blow away the peers.json file if present, since the\n\t\t\t// peers.info sentinel wasn't there.\n\t\t\tif _, err := os.Stat(peersFile); err == nil {\n\t\t\t\tif err := os.Remove(peersFile); err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to delete peers.json, please delete manually (see peers.info for details): %v\", err)\n\t\t\t\t}\n\t\t\t\ts.logger.Info(\"deleted peers.json file (see peers.info for details)\")\n\t\t\t}\n\t\t} else if _, err := os.Stat(peersFile); err == nil {\n\t\t\ts.logger.Info(\"found peers.json file, recovering Raft configuration...\")\n\t\t\tvar configuration raft.Configuration\n\t\t\tif s.config.RaftConfig.ProtocolVersion < 3 {\n\t\t\t\tconfiguration, err = raft.ReadPeersJSON(peersFile)\n\t\t\t} else {\n\t\t\t\tconfiguration, err = raft.ReadConfigJSON(peersFile)\n\t\t\t}\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed to parse peers.json: %v\", err)\n\t\t\t}\n\t\t\ttmpFsm, err := NewFSM(fsmConfig)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed to make temp FSM: %v\", err)\n\t\t\t}","sourceCodeStart":1526,"sourceCodeEnd":1562,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/server.go#L1526-L1562","documentation":"During Raft peer recovery (started when a peers.info sentinel is absent but peers.json exists as a legacy recovery file), Nomad must remove the stale peers.json. If os.Remove fails (typically a filesystem permission or I/O problem), server startup aborts with this error, telling the operator to delete the file manually. The 'see peers.info for details' reference points to Nomad's peers.info documentation sentinel created next to peers.json.","triggerScenarios":"Starting nomad server when peers.json exists but peers.info does not (operator-initiated peer recovery path) and os.Remove(peersFile) returns an error — e.g. read-only data dir, ownership mismatch, or disk error.","commonSituations":"Operator manually created peers.json to recover a lost quorum but ran Nomad as a different user than the file owner; data directory mounted read-only; container image with wrong volume permissions; out-of-disk / corrupted filesystem.","solutions":["Delete peers.json manually: rm <data_dir>/peer_id/../peers.json (the path logged by Nomad) and restart the agent","Fix filesystem permissions on the Nomad data dir (chown/chmod to the user Nomad runs as) and restart","If the data dir is read-only, remount it read-write or move data_dir to writable storage","Check dmesg/journal for underlying disk I/O errors and repair the filesystem"],"exampleFix":"// before\nsudo systemctl start nomad   # fails: failed to delete peers.json\n// after\nsudo rm /var/nomad/data/peers.json\nsudo chown -R nomad:nomad /var/nomad/data\nsudo systemctl start nomad","handlingStrategy":"validation","validationCode":"// Before starting Nomad recovery, verify the file is removable\nf=\"$(nomad config 2>/dev/null | grep -oP 'data_dir = \"\\K[^\"]*')/peers.json\"\nif [ -f \"$f\" ]; then rm \"$f\" || { echo \"cannot remove $f — fix perms/mount\"; exit 1; }; fi","typeGuard":null,"tryCatchPattern":"try {\n  startNomadServer();\n} catch (e) {\n  if (String(e).includes('failed to delete peers.json')) {\n    fs.unlinkSync(path.join(dataDir, 'peers.json')); // delete manually\n    startNomadServer();\n  } else throw e;\n}","preventionTips":["Run Nomad as the same user that owns the data dir and peers.json","Keep the data dir on a writable mount; monitor disk health","Remove peers.json manually right after creating it for recovery, before restarting the agent"],"tags":["raft","filesystem","recovery","permissions"],"backgroundTag":"raft-peers-json-recovery","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}