hashicorp/nomad · error
failed to close WAL store
Error message
failed to close WAL store: %w
What it means
Once verification passes, both stores must be closed before the BoltDB file is renamed. This error wraps a failure closing the destination WAL store (e.g. flush/sync errors while finalizing WAL segments). The BoltDB source is still closed and the WAL directory cleaned up, so raft.db remains valid and migration can be retried.
Solutions
- Read the wrapped error — for ENOSPC free space on the raft volume before retrying.
- Avoid hosting raft data on NFS/network filesystems; use local block storage with working fsync.
- Check storage health (dmesg) if EIO is wrapped.
- Re-run MigrateToWAL; cleanupWAL removes the partially finalized WAL directory so retry is safe.
Defensive patterns
Strategy: retry
Validate before calling
usage, _ := disk.Usage(raftDir)
if usage.Free < 512*1024*1024 {
return fmt.Errorf("insufficient free space for WAL finalization")
} Try / catch
err := raftutil.MigrateToWAL(ctx, raftDir, progress)
if err != nil && strings.Contains(err.Error(), "failed to close WAL store") {
os.RemoveAll(filepath.Join(raftDir, "wal"))
// retry once storage issue (ENOSPC/EIO) is resolved
err = raftutil.MigrateToWAL(ctx, raftDir, progress)
} Prevention
- Keep ample free space through the entire migration, including finalization.
- Avoid NFS/network filesystems for raft data (fsync is unreliable).
- Fix underlying storage errors before re-running.
- Always remove leftover wal/ dirs before a retry (cleanup usually does it).
When it happens
Trigger: dst.Close() returns an error: the WAL backend fails to flush or fsync buffered data on close (ENOSPC, EIO), an internal raft-wal close failure, or the store already closed/corrupted.
Common situations: Disk filled exactly at finalization; underlying storage failing fsync (network filesystems, failing disk); NFS or other non-POSIX filesystems with unreliable fsync semantics.
Related errors
- failed to close BoltDB store
- failed to sync snapshot
- failed to sync temp snapshot
- can't seek to offset
- cannot write nomad binary for install
AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04).
Data as JSON: /api/errors/80635ae210ba5d7b.
Report an issue: GitHub.
Appendix: source
Thrown at helper/raftutil/migrate.go:136
src.Close()
cleanupWAL(walDir)
return fmt.Errorf("failed to copy stable store: %w", err)
}
// Verify data integrity before finalizing.
sendProgress(progress, "verifying migrated data")
if err := verifyMigration(src, dst); err != nil {
dst.Close()
src.Close()
cleanupWAL(walDir)
return fmt.Errorf("data verification failed: %w", err)
}
// Close both stores before renaming files.
if err := dst.Close(); err != nil {
src.Close()
cleanupWAL(walDir)
return fmt.Errorf("failed to close WAL store: %w", err)
}
if err := src.Close(); err != nil {
cleanupWAL(walDir)
return fmt.Errorf("failed to close BoltDB store: %w", err)
}
// Rename the old BoltDB file to preserve it as a backup with timestamp.
timestamp := time.Now().Format("20060102-150405")
backupPath := fmt.Sprintf("%s.migrated.%s", boltPath, timestamp)
if err := os.Rename(boltPath, backupPath); err != nil {
return fmt.Errorf("migration succeeded but failed to rename %s to %s: %w",
boltPath, backupPath, err)
}
sendProgress(progress, fmt.Sprintf("migration complete; old BoltDB file preserved at %s", backupPath))
return nil
}View on GitHub (pinned to 482b49bf1a)