{"record":{"id":"5ef81f55d655e75b","repo":"hashicorp/nomad","slug":"failed-to-open-wal-log-store-v","errorCode":null,"errorMessage":"failed to open WAL log store: %v","messagePattern":"failed to open WAL log store: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/server.go","lineNumber":1459,"sourceCode":"\t\tcase LogStoreBackendWAL:\n\t\t\t// Check for an existing BoltDB store that needs migration.\n\t\t\tboltPath := filepath.Join(path, \"raft.db\")\n\t\t\tif _, statErr := os.Stat(boltPath); statErr == nil {\n\t\t\t\treturn fmt.Errorf(\n\t\t\t\t\t\"existing BoltDB raft store found at %s; \"+\n\t\t\t\t\t\t\"run 'nomad operator raft migrate-backend %s' while the server \"+\n\t\t\t\t\t\t\"is stopped to migrate to the WAL backend, then start the server again\",\n\t\t\t\t\tboltPath, s.config.DataDir)\n\t\t\t}\n\n\t\t\twalDir := filepath.Join(path, \"wal\")\n\t\t\tif err := ensurePath(walDir, true); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to create WAL directory: %v\", err)\n\t\t\t}\n\n\t\t\twalStore, walErr := s.openRaftWAL(walDir)\n\t\t\tif walErr != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to open WAL log store: %v\", walErr)\n\t\t\t}\n\t\t\tstore = walStore\n\n\t\tcase LogStoreBackendBoltDB:\n\t\t\t// Create the BoltDB backend, with NoFreelistSync option\n\t\t\tnoFreelistSync := false\n\t\t\tif s.config.RaftLogStoreConfig != nil {\n\t\t\t\tnoFreelistSync = s.config.RaftLogStoreConfig.BoltDBNoFreelistSync\n\t\t\t}\n\n\t\t\tboltStore, boltErr := raftboltdb.New(raftboltdb.Options{\n\t\t\t\tPath:   filepath.Join(path, \"raft.db\"),\n\t\t\t\tNoSync: false, // fsync each log write\n\t\t\t\tBoltOptions: &bbolt.Options{\n\t\t\t\t\tNoFreelistSync: noFreelistSync,\n\t\t\t\t},\n\t\t\t\tMsgpackUseNewTimeFormat: true,\n\t\t\t})","sourceCodeStart":1441,"sourceCodeEnd":1477,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/server.go#L1441-L1477","documentation":"After creating the WAL directory, Nomad opens the WAL-backed raft log store with s.openRaftWAL(walDir). Any error opening/initializing the WAL (corruption, lock contention, I/O failure) is wrapped in this error and aborts raft setup.","triggerScenarios":"setupRaft(): openRaftWAL(walDir) returns an error — e.g. corrupted WAL segments in <raft path>/wal, another process holding locks, or disk I/O errors.","commonSituations":"Unclean shutdown leaving corrupt WAL segments; two nomad server processes started against the same data_dir; disk failure/ENOSPC during WAL open.","solutions":["Ensure no other nomad process is using the same data_dir (ps aux | grep nomad), then restart","Check the underlying error in %v: if WAL segments are corrupt, restore from backup/snapshot and follow HashiCorp's raft recovery procedures","Fix disk issues (space, hardware) indicated by the wrapped error"],"exampleFix":"# before: two servers on same data dir\n$ nomad agent -server -data-dir /var/lib/nomad  # second instance fails opening WAL\n# after\n$ pkill -f 'nomad agent'   # stop the stale instance\n$ nomad agent -server -data-dir /var/lib/nomad","handlingStrategy":"retry","validationCode":"// No pre-call check can detect WAL corruption; verify no other process holds the data dir.\nif pid := lockHeld(dataDir); pid != 0 {\n    return fmt.Errorf(\"data_dir in use by pid %d\", pid)\n}","typeGuard":null,"tryCatchPattern":"if err := server.Start(); err != nil {\n    if strings.Contains(err.Error(), \"failed to open WAL log store\") {\n        logger.Error(\"WAL open failed; check for stale process/corruption\", \"err\", err)\n        // restart once after confirming single instance; else restore from snapshot\n    }\n    return err\n}","preventionTips":["Use process supervision to guarantee a single server instance per data_dir","Back up data_dir and maintain raft snapshots for recovery","Monitor disk health/space to avoid I/O failures during WAL open"],"tags":["storage","raft","wal","nomad"],"backgroundTag":"log-store-open-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}