hashicorp/nomad · error

recovery failed to make temp FSM: %v

Error message

recovery failed to make temp FSM: %v

What it means

As part of raft.RecoverCluster, Nomad builds a throwaway FSM (NewFSM) to replay/apply the recovered Raft configuration against state snapshots. If constructing that temporary FSM fails — typically because state store paths or the FSM config (log store/snapshot refs in fsmConfig) are bad — startup aborts with this error.

Source

Thrown at nomad/server.go:1561

				if err := os.Remove(peersFile); err != nil {
					return fmt.Errorf("failed to delete peers.json, please delete manually (see peers.info for details): %v", err)
				}
				s.logger.Info("deleted peers.json file (see peers.info for details)")
			}
		} else if _, err := os.Stat(peersFile); err == nil {
			s.logger.Info("found peers.json file, recovering Raft configuration...")
			var configuration raft.Configuration
			if s.config.RaftConfig.ProtocolVersion < 3 {
				configuration, err = raft.ReadPeersJSON(peersFile)
			} else {
				configuration, err = raft.ReadConfigJSON(peersFile)
			}
			if err != nil {
				return fmt.Errorf("recovery failed to parse peers.json: %v", err)
			}
			tmpFsm, err := NewFSM(fsmConfig)
			if err != nil {
				return fmt.Errorf("recovery failed to make temp FSM: %v", err)
			}
			if err := raft.RecoverCluster(s.config.RaftConfig, tmpFsm,
				log, stable, snap, trans, configuration); err != nil {
				return fmt.Errorf("recovery failed: %v", err)
			}
			if err := os.Remove(peersFile); err != nil {
				return fmt.Errorf("recovery failed to delete peers.json, please delete manually (see peers.info for details): %v", err)
			}
			s.logger.Info("deleted peers.json file after successful recovery")
		}
	}

	// If we are a single server cluster and the state is clean then we can
	// bootstrap now.
	if s.isSingleServerCluster() {
		hasState, err := raft.HasExistingState(log, stable, snap)
		if err != nil {
			return err

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the data_dir contents are intact and readable (raft/ dir with logs and snapshots present)
2. Re-run recovery on a full copy of the original data dir so snapshots/logs are consistent
3. Upgrade Nomad to the latest patch release — NewFSM failures here are rare and version-sensitive
4. If state is disposable, wipe the data dir and rebootstrap the cluster instead of recovering

Example fix

// before
nomad server start  # recovery failed to make temp FSM
// after
sudo systemctl stop nomad
cp -a /var/nomad/data /var/nomad/data.backup
# re-attempt recovery with pristine peers.json against the backup copy
sudo systemctl start nomad

// Run recovery on a pristine backup copy and verify raft state exists first
const raftDir = dataDir + '/raft';
if (!fs.existsSync(raftDir) || fs.readdirSync(raftDir).length === 0)
  throw new Error('raft state missing — RecoverCluster/NewFSM cannot run on empty or moved state');

try {
  startNomadServer();
} catch (e) {
  if (String(e).includes('recovery failed to make temp FSM')) {
    // restore data dir from backup and retry
    restoreDataDirFromBackup();
    startNomadServer();
  } else throw e;
}

When it happens

Trigger: peers.json recovery path where NewFSM(fsmConfig) returns an error, e.g. invalid config values passed into fsmConfig, or failure allocating the FSM's internal state store for the recovery.

Common situations: Corrupted or migrated data_dir state; running recovery on a copy of the data dir where paths were moved; resource exhaustion preventing state store creation; Nomad binary/config mismatch during an upgrade-recovery attempt.

Related errors

• Failed to create redacted snapshot: %v
• Raft error when restoring snapshot: %v
• failed adding job to periodic dispatcher: %v
• periodicDispatcher.Remove failed: %w
• index update failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8a815eb8e854dee9. Report an issue: GitHub.