{"record":{"id":"e63322f015996909","repo":"hashicorp/nomad","slug":"failed-to-copy-stable-store-w","errorCode":null,"errorMessage":"failed to copy stable store: %w","messagePattern":"failed to copy stable store: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"helper/raftutil/migrate.go","lineNumber":120,"sourceCode":"\tlogProgress := make(chan string, 64)\n\twg.Add(1)\n\tgo drainProgress(logProgress, progress, &wg)\n\tif err := migrate.CopyLogs(ctx, dst, src, migrateBatchBytes, logProgress); err != nil {\n\t\tdst.Close()\n\t\tsrc.Close()\n\t\tcleanupWAL(walDir)\n\t\treturn fmt.Errorf(\"failed to copy logs: %w\", err)\n\t}\n\n\t// Copy stable store keys.\n\tstableProgress := make(chan string, 64)\n\twg.Add(1)\n\tgo drainProgress(stableProgress, progress, &wg)\n\tif err := migrate.CopyStable(ctx, dst, src, nil, nil, stableProgress); err != nil {\n\t\tdst.Close()\n\t\tsrc.Close()\n\t\tcleanupWAL(walDir)\n\t\treturn fmt.Errorf(\"failed to copy stable store: %w\", err)\n\t}\n\n\t// Verify data integrity before finalizing.\n\tsendProgress(progress, \"verifying migrated data\")\n\tif err := verifyMigration(src, dst); err != nil {\n\t\tdst.Close()\n\t\tsrc.Close()\n\t\tcleanupWAL(walDir)\n\t\treturn fmt.Errorf(\"data verification failed: %w\", err)\n\t}\n\n\t// Close both stores before renaming files.\n\tif err := dst.Close(); err != nil {\n\t\tsrc.Close()\n\t\tcleanupWAL(walDir)\n\t\treturn fmt.Errorf(\"failed to close WAL store: %w\", err)\n\t}\n","sourceCodeStart":102,"sourceCodeEnd":138,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/raftutil/migrate.go#L102-L138","documentation":"After logs, migrate.CopyStable copies all stable-store key/value pairs (term, last vote, etc.) from BoltDB to the WAL store. This error wraps any failure during that copy, including context cancellation. On failure both stores are closed and the WAL directory is removed; raft.db is untouched.","triggerScenarios":"migrate.CopyStable(ctx, dst, src, nil, nil, stableProgress) returns an error: write failure into the WAL stable store (disk full, I/O error), read failure from the BoltDB stable bucket, or ctx cancelled during the copy.","commonSituations":"Disk exhaustion partway through migration (logs copied OK, stable copy fails); context deadline exceeded on slow storage; corrupted BoltDB stable bucket after a crash.","solutions":["Inspect the wrapped error — for ENOSPC free space on the raft volume and retry.","Re-run with a context that is not cancelled mid-migration (no short timeout).","Run bolt check against raft.db if a read/corruption error is wrapped.","Simply retry MigrateToWAL; cleanup guarantees a fresh start."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if boltInfo, err := os.Stat(filepath.Join(raftDir, \"raft.db\")); err == nil {\n    _ = boltInfo\n}\n// ensure >= raft.db size + 512 MiB free space before starting","typeGuard":null,"tryCatchPattern":"err := raftutil.MigrateToWAL(ctx, raftDir, progress)\nif err != nil && strings.Contains(err.Error(), \"failed to copy stable store\") {\n    if errors.Is(errors.Unwrap(errors.Unwrap(err)), syscall.ENOSPC) {\n        // free space, then retry; WAL dir was cleaned up automatically\n    }\n}","preventionTips":["Keep the migration context alive for the whole run (no aggressive timeouts).","Reserve disk headroom: raft.db size + 512 MiB minimum.","Verify source BoltDB integrity before migrating a crash-affected node.","Always retry from a clean state; MigrateToWAL removes partial WAL dirs."],"tags":["migration","raft","stable-store","io"],"backgroundTag":"stable-copy-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}