What causes "failed to copy stable store: %w" in hashicorp/nomad?

Inspect the wrapped error — for ENOSPC free space on the raft volume and retry.

failed to copy stable store: %w

Error message

failed to copy stable store: %w

What it means

After logs, migrate.CopyStable copies all stable-store key/value pairs (term, last vote, etc.) from BoltDB to the WAL store. This error wraps any failure during that copy, including context cancellation. On failure both stores are closed and the WAL directory is removed; raft.db is untouched.

Source

Thrown at helper/raftutil/migrate.go:120

	logProgress := make(chan string, 64)
	wg.Add(1)
	go drainProgress(logProgress, progress, &wg)
	if err := migrate.CopyLogs(ctx, dst, src, migrateBatchBytes, logProgress); err != nil {
		dst.Close()
		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("failed to copy logs: %w", err)
	}

	// Copy stable store keys.
	stableProgress := make(chan string, 64)
	wg.Add(1)
	go drainProgress(stableProgress, progress, &wg)
	if err := migrate.CopyStable(ctx, dst, src, nil, nil, stableProgress); err != nil {
		dst.Close()
		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("failed to copy stable store: %w", err)
	}

	// Verify data integrity before finalizing.
	sendProgress(progress, "verifying migrated data")
	if err := verifyMigration(src, dst); err != nil {
		dst.Close()
		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("data verification failed: %w", err)
	}

	// Close both stores before renaming files.
	if err := dst.Close(); err != nil {
		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("failed to close WAL store: %w", err)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

Inspect the wrapped error — for ENOSPC free space on the raft volume and retry.
Re-run with a context that is not cancelled mid-migration (no short timeout).
Run bolt check against raft.db if a read/corruption error is wrapped.
Simply retry MigrateToWAL; cleanup guarantees a fresh start.

Defensive patterns

Strategy: try-catch

Validate before calling

if boltInfo, err := os.Stat(filepath.Join(raftDir, "raft.db")); err == nil {
    _ = boltInfo
}
// ensure >= raft.db size + 512 MiB free space before starting

Try / catch

err := raftutil.MigrateToWAL(ctx, raftDir, progress)
if err != nil && strings.Contains(err.Error(), "failed to copy stable store") {
    if errors.Is(errors.Unwrap(errors.Unwrap(err)), syscall.ENOSPC) {
        // free space, then retry; WAL dir was cleaned up automatically
    }
}

Prevention

Keep the migration context alive for the whole run (no aggressive timeouts).
Reserve disk headroom: raft.db size + 512 MiB minimum.
Verify source BoltDB integrity before migrating a crash-affected node.
Always retry from a clean state; MigrateToWAL removes partial WAL dirs.

When it happens

Trigger: migrate.CopyStable(ctx, dst, src, nil, nil, stableProgress) returns an error: write failure into the WAL stable store (disk full, I/O error), read failure from the BoltDB stable bucket, or ctx cancelled during the copy.

Common situations: Disk exhaustion partway through migration (logs copied OK, stable copy fails); context deadline exceeded on slow storage; corrupted BoltDB stable bucket after a crash.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e63322f015996909. Report an issue: GitHub.