hashicorp/nomad · error

failed to close WAL store: %w

Error message

failed to close WAL store: %w

What it means

Once verification passes, both stores must be closed before the BoltDB file is renamed. This error wraps a failure closing the destination WAL store (e.g. flush/sync errors while finalizing WAL segments). The BoltDB source is still closed and the WAL directory cleaned up, so raft.db remains valid and migration can be retried.

Source

Thrown at helper/raftutil/migrate.go:136

		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("failed to copy stable store: %w", err)
	}

	// Verify data integrity before finalizing.
	sendProgress(progress, "verifying migrated data")
	if err := verifyMigration(src, dst); err != nil {
		dst.Close()
		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("data verification failed: %w", err)
	}

	// Close both stores before renaming files.
	if err := dst.Close(); err != nil {
		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("failed to close WAL store: %w", err)
	}

	if err := src.Close(); err != nil {
		cleanupWAL(walDir)
		return fmt.Errorf("failed to close BoltDB store: %w", err)
	}

	// Rename the old BoltDB file to preserve it as a backup with timestamp.
	timestamp := time.Now().Format("20060102-150405")
	backupPath := fmt.Sprintf("%s.migrated.%s", boltPath, timestamp)
	if err := os.Rename(boltPath, backupPath); err != nil {
		return fmt.Errorf("migration succeeded but failed to rename %s to %s: %w",
			boltPath, backupPath, err)
	}

	sendProgress(progress, fmt.Sprintf("migration complete; old BoltDB file preserved at %s", backupPath))
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped error — for ENOSPC free space on the raft volume before retrying.
2. Avoid hosting raft data on NFS/network filesystems; use local block storage with working fsync.
3. Check storage health (dmesg) if EIO is wrapped.
4. Re-run MigrateToWAL; cleanupWAL removes the partially finalized WAL directory so retry is safe.

usage, _ := disk.Usage(raftDir)
if usage.Free < 512*1024*1024 {
    return fmt.Errorf("insufficient free space for WAL finalization")
}

err := raftutil.MigrateToWAL(ctx, raftDir, progress)
if err != nil && strings.Contains(err.Error(), "failed to close WAL store") {
    os.RemoveAll(filepath.Join(raftDir, "wal"))
    // retry once storage issue (ENOSPC/EIO) is resolved
    err = raftutil.MigrateToWAL(ctx, raftDir, progress)
}

When it happens

Trigger: dst.Close() returns an error: the WAL backend fails to flush or fsync buffered data on close (ENOSPC, EIO), an internal raft-wal close failure, or the store already closed/corrupted.

Common situations: Disk filled exactly at finalization; underlying storage failing fsync (network filesystems, failing disk); NFS or other non-POSIX filesystems with unreliable fsync semantics.

Related errors

• failed to close BoltDB store: %w
• failed to sync snapshot: %v
• failed to sync temp snapshot: %v
• unable to read rooted allocation directory
• can't seek to offset %d: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/80635ae210ba5d7b. Report an issue: GitHub.