hashicorp/nomad · error

failed to create WAL directory: %v

Error message

failed to create WAL directory: %v

What it means

When the WAL raft log store backend is selected, Nomad creates <raft path>/wal via ensurePath before opening the WAL store. Failure to create this directory (permissions, read-only FS, path conflicts) aborts startup with this error.

Source

Thrown at nomad/server.go:1454

			backend = s.config.RaftLogStoreConfig.Backend
		}

		var store raftBackend
		switch backend {
		case LogStoreBackendWAL:
			// Check for an existing BoltDB store that needs migration.
			boltPath := filepath.Join(path, "raft.db")
			if _, statErr := os.Stat(boltPath); statErr == nil {
				return fmt.Errorf(
					"existing BoltDB raft store found at %s; "+
						"run 'nomad operator raft migrate-backend %s' while the server "+
						"is stopped to migrate to the WAL backend, then start the server again",
					boltPath, s.config.DataDir)
			}

			walDir := filepath.Join(path, "wal")
			if err := ensurePath(walDir, true); err != nil {
				return fmt.Errorf("failed to create WAL directory: %v", err)
			}

			walStore, walErr := s.openRaftWAL(walDir)
			if walErr != nil {
				return fmt.Errorf("failed to open WAL log store: %v", walErr)
			}
			store = walStore

		case LogStoreBackendBoltDB:
			// Create the BoltDB backend, with NoFreelistSync option
			noFreelistSync := false
			if s.config.RaftLogStoreConfig != nil {
				noFreelistSync = s.config.RaftLogStoreConfig.BoltDBNoFreelistSync
			}

			boltStore, boltErr := raftboltdb.New(raftboltdb.Options{
				Path:   filepath.Join(path, "raft.db"),
				NoSync: false, // fsync each log write

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Fix data_dir permissions/ownership so the nomad process can create directories
2. Check for a regular file named 'wal' in <data_dir>/raft and remove/rename it
3. Verify the filesystem is writable and has free space

Example fix

# before: cannot create /var/lib/nomad/raft/wal
$ ls -ld /var/lib/nomad  # owned by root
# after
$ sudo chown -R nomad:nomad /var/lib/nomad
$ sudo -u nomad nomad agent -server

wal := filepath.Join(dataDir, "raft", "wal")
if fi, err := os.Stat(wal); err == nil && !fi.IsDir() {
    return fmt.Errorf("%s exists and is not a directory", wal)
}
if err := os.MkdirAll(wal, 0700); err != nil {
    return fmt.Errorf("cannot create wal dir: %w", err)
}

if err := server.Start(); err != nil {
    if strings.Contains(err.Error(), "failed to create WAL directory") {
        logger.Error("fix data_dir writability / stray 'wal' file")
    }
    return err
}

When it happens

Trigger: setupRaft(): ensurePath(filepath.Join(path, "wal"), true) returns an error because the wal directory cannot be created under the raft data path.

Common situations: data_dir not writable by the nomad user; read-only container filesystem; a file named 'wal' already exists in the raft directory blocking mkdir; disk full or ENOSPC.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

• failed to write peers.info file: %v
• failed to find raft store in %v: %v
• no raft store (raft.db or wal/) found in %s
• failed to write Raft version file: %v
• existing BoltDB raft store found at %s; run 'nomad operator

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/1645649b13199966. Report an issue: GitHub.