hashicorp/nomad · error

failed to delete peers.json, please delete manually (see pee

Error message

failed to delete peers.json, please delete manually (see peers.info for details): %v

What it means

During Raft peer recovery (started when a peers.info sentinel is absent but peers.json exists as a legacy recovery file), Nomad must remove the stale peers.json. If os.Remove fails (typically a filesystem permission or I/O problem), server startup aborts with this error, telling the operator to delete the file manually. The 'see peers.info for details' reference points to Nomad's peers.info documentation sentinel created next to peers.json.

Source

Thrown at nomad/server.go:1544

		// For an existing cluster being upgraded to the new version of
		// Raft, we almost never want to run recovery based on the old
		// peers.json file. We create a peers.info file with a helpful
		// note about where peers.json went, and use that as a sentinel
		// to avoid ingesting the old one that first time (if we have to
		// create the peers.info file because it's not there, we also
		// blow away any existing peers.json file).
		peersFile := filepath.Join(path, "peers.json")
		peersInfoFile := filepath.Join(path, "peers.info")
		if _, err := os.Stat(peersInfoFile); os.IsNotExist(err) {
			if err := os.WriteFile(peersInfoFile, []byte(peersInfoContent), 0644); err != nil {
				return fmt.Errorf("failed to write peers.info file: %v", err)
			}

			// Blow away the peers.json file if present, since the
			// peers.info sentinel wasn't there.
			if _, err := os.Stat(peersFile); err == nil {
				if err := os.Remove(peersFile); err != nil {
					return fmt.Errorf("failed to delete peers.json, please delete manually (see peers.info for details): %v", err)
				}
				s.logger.Info("deleted peers.json file (see peers.info for details)")
			}
		} else if _, err := os.Stat(peersFile); err == nil {
			s.logger.Info("found peers.json file, recovering Raft configuration...")
			var configuration raft.Configuration
			if s.config.RaftConfig.ProtocolVersion < 3 {
				configuration, err = raft.ReadPeersJSON(peersFile)
			} else {
				configuration, err = raft.ReadConfigJSON(peersFile)
			}
			if err != nil {
				return fmt.Errorf("recovery failed to parse peers.json: %v", err)
			}
			tmpFsm, err := NewFSM(fsmConfig)
			if err != nil {
				return fmt.Errorf("recovery failed to make temp FSM: %v", err)
			}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Delete peers.json manually: rm <data_dir>/peer_id/../peers.json (the path logged by Nomad) and restart the agent
2. Fix filesystem permissions on the Nomad data dir (chown/chmod to the user Nomad runs as) and restart
3. If the data dir is read-only, remount it read-write or move data_dir to writable storage
4. Check dmesg/journal for underlying disk I/O errors and repair the filesystem

Example fix

// before
sudo systemctl start nomad   # fails: failed to delete peers.json
// after
sudo rm /var/nomad/data/peers.json
sudo chown -R nomad:nomad /var/nomad/data
sudo systemctl start nomad

// Before starting Nomad recovery, verify the file is removable
f="$(nomad config 2>/dev/null | grep -oP 'data_dir = "\K[^"]*')/peers.json"
if [ -f "$f" ]; then rm "$f" || { echo "cannot remove $f — fix perms/mount"; exit 1; }; fi

try {
  startNomadServer();
} catch (e) {
  if (String(e).includes('failed to delete peers.json')) {
    fs.unlinkSync(path.join(dataDir, 'peers.json')); // delete manually
    startNomadServer();
  } else throw e;
}

When it happens

Trigger: Starting nomad server when peers.json exists but peers.info does not (operator-initiated peer recovery path) and os.Remove(peersFile) returns an error — e.g. read-only data dir, ownership mismatch, or disk error.

Common situations: Operator manually created peers.json to recover a lost quorum but ran Nomad as a different user than the file owner; data directory mounted read-only; container image with wrong volume permissions; out-of-disk / corrupted filesystem.

Related errors

• recovery failed to delete peers.json, please delete manually
• failed to open snapshot dir: %v
• failed to stat raft store %s: %v
• plugin not executable
• failed to snapshot %s: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/3af4fec5f03b0ddc. Report an issue: GitHub.