hashicorp/nomad · error

recovery failed to delete peers.json, please delete manually

Error message

recovery failed to delete peers.json, please delete manually (see peers.info for details): %v

What it means

After raft.RecoverCluster succeeds, Nomad deletes the now-consumed peers.json. If os.Remove fails at this point the recovery itself worked, but the stale file remains and would trigger another recovery pass on next start, so startup aborts and asks the operator to remove it manually.

Source

Thrown at nomad/server.go:1568

			var configuration raft.Configuration
			if s.config.RaftConfig.ProtocolVersion < 3 {
				configuration, err = raft.ReadPeersJSON(peersFile)
			} else {
				configuration, err = raft.ReadConfigJSON(peersFile)
			}
			if err != nil {
				return fmt.Errorf("recovery failed to parse peers.json: %v", err)
			}
			tmpFsm, err := NewFSM(fsmConfig)
			if err != nil {
				return fmt.Errorf("recovery failed to make temp FSM: %v", err)
			}
			if err := raft.RecoverCluster(s.config.RaftConfig, tmpFsm,
				log, stable, snap, trans, configuration); err != nil {
				return fmt.Errorf("recovery failed: %v", err)
			}
			if err := os.Remove(peersFile); err != nil {
				return fmt.Errorf("recovery failed to delete peers.json, please delete manually (see peers.info for details): %v", err)
			}
			s.logger.Info("deleted peers.json file after successful recovery")
		}
	}

	// If we are a single server cluster and the state is clean then we can
	// bootstrap now.
	if s.isSingleServerCluster() {
		hasState, err := raft.HasExistingState(log, stable, snap)
		if err != nil {
			return err
		}
		if !hasState {
			configuration := raft.Configuration{
				Servers: []raft.Server{
					{
						ID:      s.config.RaftConfig.LocalID,
						Address: trans.LocalAddr(),

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Delete peers.json manually (path is in the log line) and restart Nomad — recovery already succeeded
2. Check the mount is not read-only (mount | grep <data_dir>) and remount read-write
3. Fix ownership/ACLs on the data dir for the Nomad service user
4. Remove any cleanup automation touching peers.json while Nomad starts

Example fix

// before
journalctl -u nomad  # recovery failed to delete peers.json ...
// after
sudo rm /var/nomad/data/peers.json
sudo systemctl restart nomad

// Ensure the recovery file is deletable before kicking off recovery
const f = dataDir + '/peers.json';
if (fs.existsSync(f)) fs.accessSync(f, fs.constants.W_OK); // throws if not writable/deletable

try {
  startNomadServer();
} catch (e) {
  if (String(e).includes('recovery failed to delete peers.json')) {
    fs.rmSync(dataDir + '/peers.json', { force: true }); // safe: recovery already succeeded
    startNomadServer();
  } else throw e;
}

When it happens

Trigger: Immediately after successful RecoverCluster, os.Remove(peersFile) errors — file deleted out from under the process, read-only filesystem, or permission change during startup.

Common situations: Filesystem flipped to read-only mid-startup (e.g. LVM/RAID errors); another process or cleanup job removed/chmod'ed the file; data dir mounted with restrictive ACLs on shared storage.

Related errors

• failed to delete peers.json, please delete manually (see pee
• failed to open snapshot dir: %v
• failed to stat raft store %s: %v
• plugin not executable
• failed to snapshot %s: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/930277cded72dd63. Report an issue: GitHub.