{"record":{"id":"8a815eb8e854dee9","repo":"hashicorp/nomad","slug":"recovery-failed-to-make-temp-fsm-v","errorCode":null,"errorMessage":"recovery failed to make temp FSM: %v","messagePattern":"recovery failed to make temp FSM: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/server.go","lineNumber":1561,"sourceCode":"\t\t\t\tif err := os.Remove(peersFile); err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to delete peers.json, please delete manually (see peers.info for details): %v\", err)\n\t\t\t\t}\n\t\t\t\ts.logger.Info(\"deleted peers.json file (see peers.info for details)\")\n\t\t\t}\n\t\t} else if _, err := os.Stat(peersFile); err == nil {\n\t\t\ts.logger.Info(\"found peers.json file, recovering Raft configuration...\")\n\t\t\tvar configuration raft.Configuration\n\t\t\tif s.config.RaftConfig.ProtocolVersion < 3 {\n\t\t\t\tconfiguration, err = raft.ReadPeersJSON(peersFile)\n\t\t\t} else {\n\t\t\t\tconfiguration, err = raft.ReadConfigJSON(peersFile)\n\t\t\t}\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed to parse peers.json: %v\", err)\n\t\t\t}\n\t\t\ttmpFsm, err := NewFSM(fsmConfig)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed to make temp FSM: %v\", err)\n\t\t\t}\n\t\t\tif err := raft.RecoverCluster(s.config.RaftConfig, tmpFsm,\n\t\t\t\tlog, stable, snap, trans, configuration); err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed: %v\", err)\n\t\t\t}\n\t\t\tif err := os.Remove(peersFile); err != nil {\n\t\t\t\treturn fmt.Errorf(\"recovery failed to delete peers.json, please delete manually (see peers.info for details): %v\", err)\n\t\t\t}\n\t\t\ts.logger.Info(\"deleted peers.json file after successful recovery\")\n\t\t}\n\t}\n\n\t// If we are a single server cluster and the state is clean then we can\n\t// bootstrap now.\n\tif s.isSingleServerCluster() {\n\t\thasState, err := raft.HasExistingState(log, stable, snap)\n\t\tif err != nil {\n\t\t\treturn err","sourceCodeStart":1543,"sourceCodeEnd":1579,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/server.go#L1543-L1579","documentation":"As part of raft.RecoverCluster, Nomad builds a throwaway FSM (NewFSM) to replay/apply the recovered Raft configuration against state snapshots. If constructing that temporary FSM fails — typically because state store paths or the FSM config (log store/snapshot refs in fsmConfig) are bad — startup aborts with this error.","triggerScenarios":"peers.json recovery path where NewFSM(fsmConfig) returns an error, e.g. invalid config values passed into fsmConfig, or failure allocating the FSM's internal state store for the recovery.","commonSituations":"Corrupted or migrated data_dir state; running recovery on a copy of the data dir where paths were moved; resource exhaustion preventing state store creation; Nomad binary/config mismatch during an upgrade-recovery attempt.","solutions":["Verify the data_dir contents are intact and readable (raft/ dir with logs and snapshots present)","Re-run recovery on a full copy of the original data dir so snapshots/logs are consistent","Upgrade Nomad to the latest patch release — NewFSM failures here are rare and version-sensitive","If state is disposable, wipe the data dir and rebootstrap the cluster instead of recovering"],"exampleFix":"// before\nnomad server start  # recovery failed to make temp FSM\n// after\nsudo systemctl stop nomad\ncp -a /var/nomad/data /var/nomad/data.backup\n# re-attempt recovery with pristine peers.json against the backup copy\nsudo systemctl start nomad","handlingStrategy":"validation","validationCode":"// Run recovery on a pristine backup copy and verify raft state exists first\nconst raftDir = dataDir + '/raft';\nif (!fs.existsSync(raftDir) || fs.readdirSync(raftDir).length === 0)\n  throw new Error('raft state missing — RecoverCluster/NewFSM cannot run on empty or moved state');","typeGuard":null,"tryCatchPattern":"try {\n  startNomadServer();\n} catch (e) {\n  if (String(e).includes('recovery failed to make temp FSM')) {\n    // restore data dir from backup and retry\n    restoreDataDirFromBackup();\n    startNomadServer();\n  } else throw e;\n}","preventionTips":["Always take a full copy of the data dir before peer-recovery operations","Keep snapshots and logs together; never move raft state across paths piecemeal","Keep Nomad patched to the latest release"],"tags":["raft","recovery","fsm","internal"],"backgroundTag":"raft-peers-json-recovery","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}