{"record":{"id":"f0f45e70310b61eb","repo":"hashicorp/nomad","slug":"failed-to-open-raft-store-v-v","errorCode":null,"errorMessage":"failed to open raft store %v: %v","messagePattern":"failed to open raft store (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"helper/raftutil/fsm.go","lineNumber":54,"sourceCode":"\tfsm   nomadFSM\n\tsnaps *raft.FileSnapshotStore\n\n\t// raft\n\tlogFirstIdx uint64\n\tlogLastIdx  uint64\n\tnextIdx     uint64\n}\n\nfunc NewFSM(p string) (*FSMHelper, error) {\n\t// Auto-detect the backend: look for wal/ directory first, then raft.db.\n\tstorePath, err := FindRaftStore(p)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to find raft store in %v: %v\", p, err)\n\t}\n\n\tstore, firstIdx, lastIdx, err := RaftStateInfo(storePath)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to open raft store %v: %v\", storePath, err)\n\t}\n\n\tlogger := hclog.L()\n\n\tsnaps, err := raft.NewFileSnapshotStoreWithLogger(p, 1000, logger)\n\tif err != nil {\n\t\tstore.Close()\n\t\treturn nil, fmt.Errorf(\"failed to open snapshot dir: %v\", err)\n\t}\n\n\tfsm, err := dummyFSM(logger)\n\tif err != nil {\n\t\tstore.Close()\n\t\treturn nil, err\n\t}\n\n\treturn &FSMHelper{\n\t\tpath:   p,","sourceCodeStart":36,"sourceCodeEnd":72,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/raftutil/fsm.go#L36-L72","documentation":"NewFSM wraps the error from RaftStateInfo when the located raft store (BoltDB or WAL) cannot be opened, e.g. corrupt file, lock contention, or unsupported/failed backend. The store path is included in the message.","triggerScenarios":"Calling NewFSM where raft.db is corrupt/truncated, another process holds the BoltDB file lock, disk I/O errors occur, or the WAL directory is unreadable/corrupt.","commonSituations":"Running migration/inspection tooling while the Nomad server is still running (BoltDB file lock held); corrupted data dir after crash or disk-full; insufficient file permissions on raft.db.","solutions":["Stop the Nomad server process before opening the store (BoltDB allows only one writer/lock holder)","Check file permissions and disk health; look at the wrapped error for 'timeout' (lock) vs corruption","Restore the data dir from a snapshot or backup if the store is corrupt (nomad operator snapshot inspect/restore)"],"exampleFix":"// before\nfsm, err := raftutil.NewFSM(dataDir) // fails if server is running\n// after\nif err := stopServerAndAcquireLock(dataDir); err != nil {\n    return fmt.Errorf(\"ensure nomad server is stopped before opening %s: %w\", dataDir, err)\n}\nfsm, err := raftutil.NewFSM(dataDir)","handlingStrategy":"validation","validationCode":"if err := probeBoltDBOpen(filepath.Join(dataDir, \"raft.db\")); err != nil {\n    return fmt.Errorf(\"stop the nomad server or fix raft.db before proceeding: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := raftutil.NewFSM(dataDir); err != nil {\n    if strings.Contains(err.Error(), \"failed to open raft store\") && strings.Contains(err.Error(), \"timeout\") {\n        return fmt.Errorf(\"raft.db locked — is the nomad server still running? %w\", err)\n    }\n    return err\n}","preventionTips":["Always stop the Nomad server before opening its raft store offline","Never share a data dir between processes; BoltDB permits a single lock holder","Maintain backups/snapshots so a corrupt store can be restored","Check disk health if open failures repeat"],"tags":["raft","boltdb","storage","lock","go"],"backgroundTag":"boltdb-open-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}