hashicorp/nomad · error

failed to create migration marker: %w

Error message

failed to create migration marker: %w

What it means

MigrateToWAL fails when it cannot write the migration marker file used to detect if a server accidentally starts mid-migration. The marker is written with mode 0600 next to the data dir before opening the source store.

Source

Thrown at helper/raftutil/migrate.go:72

			close(progress)
		}
	}()

	boltPath := filepath.Join(raftDir, "raft.db")
	walDir := filepath.Join(raftDir, "wal")
	markerPath := filepath.Join(raftDir, migrationMarkerFile)

	sendProgress(progress, "starting migration pre-flight checks")

	if err := preflightChecks(boltPath, walDir, raftDir); err != nil {
		return err
	}

	sendProgress(progress, "pre-flight checks passed")

	// Create marker file to detect if server accidentally starts during migration.
	if err := os.WriteFile(markerPath, []byte(time.Now().Format(time.RFC3339)), 0o600); err != nil {
		return fmt.Errorf("failed to create migration marker: %w", err)
	}
	defer os.Remove(markerPath) // Clean up marker on completion or failure.

	// Open the source BoltDB store.
	src, err := raftboltdb.New(raftboltdb.Options{
		Path: boltPath,
		BoltOptions: &bbolt.Options{
			Timeout: 5 * time.Second,
		},
		MsgpackUseNewTimeFormat: true,
	})
	if err != nil {
		return fmt.Errorf("failed to open BoltDB store: %w", err)
	}

	// Create the destination WAL store.
	if err := os.MkdirAll(walDir, 0o700); err != nil {
		src.Close()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Run the migration as a user with write access to the data directory (or fix ownership: chown)
2. Ensure the filesystem is mounted read-write and has free space
3. Check for MAC policies (SELinux/AppArmor) denying writes if permissions look correct

Example fix

// before
err := raftutil.MigrateToWAL(dataDir)
// after
if err := checkDirWritable(dataDir); err != nil {
    return fmt.Errorf("cannot write to %s: run migration as the nomad user", dataDir)
}
err := raftutil.MigrateToWAL(dataDir)

probe := filepath.Join(dataDir, ".write-test")
if err := os.WriteFile(probe, nil, 0o600); err != nil {
    return fmt.Errorf("data dir not writable: run as the nomad user or fix mounts: %w", err)
}
os.Remove(probe)

if err := raftutil.MigrateToWAL(dataDir); err != nil {
    if strings.Contains(err.Error(), "failed to create migration marker") {
        return fmt.Errorf("fix write permissions on %s, then remove stale markers and retry: %w", dataDir, err)
    }
    return err
}

When it happens

Trigger: Calling MigrateToWAL when the data directory is read-only, the process lacks write permission, the disk is full, or markerPath points at an unwritable location.

Common situations: Running the migration tool as a non-root user against a root-owned Nomad data dir; performing migration on a read-only mounted volume; SELinux/AppArmor blocking file creation.

Understand the failure class

Background: "Permission denied" / "Failed to write" file errors: why a library can't write its files to disk (EACCES, EPERM, ENOSPC) and how to fix them — this error's family across 43 libraries.

Related errors

• error creating directory: %w
• error setting directory permission mode: %w
• cannot create copy - %s
• failed to scan plugin directory %q: %v
• failed to open plugin directory %q: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/478e10d45ed7f0ca. Report an issue: GitHub.