hashicorp/nomad · error

failed to open WAL log store: %v

Error message

failed to open WAL log store: %v

What it means

After creating the WAL directory, Nomad opens the WAL-backed raft log store with s.openRaftWAL(walDir). Any error opening/initializing the WAL (corruption, lock contention, I/O failure) is wrapped in this error and aborts raft setup.

Source

Thrown at nomad/server.go:1459

		case LogStoreBackendWAL:
			// Check for an existing BoltDB store that needs migration.
			boltPath := filepath.Join(path, "raft.db")
			if _, statErr := os.Stat(boltPath); statErr == nil {
				return fmt.Errorf(
					"existing BoltDB raft store found at %s; "+
						"run 'nomad operator raft migrate-backend %s' while the server "+
						"is stopped to migrate to the WAL backend, then start the server again",
					boltPath, s.config.DataDir)
			}

			walDir := filepath.Join(path, "wal")
			if err := ensurePath(walDir, true); err != nil {
				return fmt.Errorf("failed to create WAL directory: %v", err)
			}

			walStore, walErr := s.openRaftWAL(walDir)
			if walErr != nil {
				return fmt.Errorf("failed to open WAL log store: %v", walErr)
			}
			store = walStore

		case LogStoreBackendBoltDB:
			// Create the BoltDB backend, with NoFreelistSync option
			noFreelistSync := false
			if s.config.RaftLogStoreConfig != nil {
				noFreelistSync = s.config.RaftLogStoreConfig.BoltDBNoFreelistSync
			}

			boltStore, boltErr := raftboltdb.New(raftboltdb.Options{
				Path:   filepath.Join(path, "raft.db"),
				NoSync: false, // fsync each log write
				BoltOptions: &bbolt.Options{
					NoFreelistSync: noFreelistSync,
				},
				MsgpackUseNewTimeFormat: true,
			})

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Ensure no other nomad process is using the same data_dir (ps aux | grep nomad), then restart
2. Check the underlying error in %v: if WAL segments are corrupt, restore from backup/snapshot and follow HashiCorp's raft recovery procedures
3. Fix disk issues (space, hardware) indicated by the wrapped error

Example fix

# before: two servers on same data dir
$ nomad agent -server -data-dir /var/lib/nomad  # second instance fails opening WAL
# after
$ pkill -f 'nomad agent'   # stop the stale instance
$ nomad agent -server -data-dir /var/lib/nomad

// No pre-call check can detect WAL corruption; verify no other process holds the data dir.
if pid := lockHeld(dataDir); pid != 0 {
    return fmt.Errorf("data_dir in use by pid %d", pid)
}

if err := server.Start(); err != nil {
    if strings.Contains(err.Error(), "failed to open WAL log store") {
        logger.Error("WAL open failed; check for stale process/corruption", "err", err)
        // restart once after confirming single instance; else restore from snapshot
    }
    return err
}

When it happens

Trigger: setupRaft(): openRaftWAL(walDir) returns an error — e.g. corrupted WAL segments in <raft path>/wal, another process holding locks, or disk I/O errors.

Common situations: Unclean shutdown leaving corrupt WAL segments; two nomad server processes started against the same data_dir; disk failure/ENOSPC during WAL open.

Related errors

• existing BoltDB raft store found at %s; run 'nomad operator
• failed to create WAL directory: %v
• failed to write peers.info file: %v
• CSI.ControllerAttachVolume: VolumeID is required
• CSI.ControllerAttachVolume: ClientCSINodeID is required

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/5ef81f55d655e75b. Report an issue: GitHub.