{"record":{"id":"80635ae210ba5d7b","repo":"hashicorp/nomad","slug":"failed-to-close-wal-store-w","errorCode":null,"errorMessage":"failed to close WAL store: %w","messagePattern":"failed to close WAL store: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"helper/raftutil/migrate.go","lineNumber":136,"sourceCode":"\t\tsrc.Close()\n\t\tcleanupWAL(walDir)\n\t\treturn fmt.Errorf(\"failed to copy stable store: %w\", err)\n\t}\n\n\t// Verify data integrity before finalizing.\n\tsendProgress(progress, \"verifying migrated data\")\n\tif err := verifyMigration(src, dst); err != nil {\n\t\tdst.Close()\n\t\tsrc.Close()\n\t\tcleanupWAL(walDir)\n\t\treturn fmt.Errorf(\"data verification failed: %w\", err)\n\t}\n\n\t// Close both stores before renaming files.\n\tif err := dst.Close(); err != nil {\n\t\tsrc.Close()\n\t\tcleanupWAL(walDir)\n\t\treturn fmt.Errorf(\"failed to close WAL store: %w\", err)\n\t}\n\n\tif err := src.Close(); err != nil {\n\t\tcleanupWAL(walDir)\n\t\treturn fmt.Errorf(\"failed to close BoltDB store: %w\", err)\n\t}\n\n\t// Rename the old BoltDB file to preserve it as a backup with timestamp.\n\ttimestamp := time.Now().Format(\"20060102-150405\")\n\tbackupPath := fmt.Sprintf(\"%s.migrated.%s\", boltPath, timestamp)\n\tif err := os.Rename(boltPath, backupPath); err != nil {\n\t\treturn fmt.Errorf(\"migration succeeded but failed to rename %s to %s: %w\",\n\t\t\tboltPath, backupPath, err)\n\t}\n\n\tsendProgress(progress, fmt.Sprintf(\"migration complete; old BoltDB file preserved at %s\", backupPath))\n\treturn nil\n}","sourceCodeStart":118,"sourceCodeEnd":154,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/raftutil/migrate.go#L118-L154","documentation":"Once verification passes, both stores must be closed before the BoltDB file is renamed. This error wraps a failure closing the destination WAL store (e.g. flush/sync errors while finalizing WAL segments). The BoltDB source is still closed and the WAL directory cleaned up, so raft.db remains valid and migration can be retried.","triggerScenarios":"dst.Close() returns an error: the WAL backend fails to flush or fsync buffered data on close (ENOSPC, EIO), an internal raft-wal close failure, or the store already closed/corrupted.","commonSituations":"Disk filled exactly at finalization; underlying storage failing fsync (network filesystems, failing disk); NFS or other non-POSIX filesystems with unreliable fsync semantics.","solutions":["Read the wrapped error — for ENOSPC free space on the raft volume before retrying.","Avoid hosting raft data on NFS/network filesystems; use local block storage with working fsync.","Check storage health (dmesg) if EIO is wrapped.","Re-run MigrateToWAL; cleanupWAL removes the partially finalized WAL directory so retry is safe."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"usage, _ := disk.Usage(raftDir)\nif usage.Free < 512*1024*1024 {\n    return fmt.Errorf(\"insufficient free space for WAL finalization\")\n}","typeGuard":null,"tryCatchPattern":"err := raftutil.MigrateToWAL(ctx, raftDir, progress)\nif err != nil && strings.Contains(err.Error(), \"failed to close WAL store\") {\n    os.RemoveAll(filepath.Join(raftDir, \"wal\"))\n    // retry once storage issue (ENOSPC/EIO) is resolved\n    err = raftutil.MigrateToWAL(ctx, raftDir, progress)\n}","preventionTips":["Keep ample free space through the entire migration, including finalization.","Avoid NFS/network filesystems for raft data (fsync is unreliable).","Fix underlying storage errors before re-running.","Always remove leftover wal/ dirs before a retry (cleanup usually does it)."],"tags":["raft-wal","close-failed","fsync","io"],"backgroundTag":"wal-close-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}