{"record":{"id":"3b8ef9fd94b87074","repo":"hashicorp/nomad","slug":"data-verification-failed-w","errorCode":null,"errorMessage":"data verification failed: %w","messagePattern":"data verification failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"helper/raftutil/migrate.go","lineNumber":129,"sourceCode":"\n\t// Copy stable store keys.\n\tstableProgress := make(chan string, 64)\n\twg.Add(1)\n\tgo drainProgress(stableProgress, progress, &wg)\n\tif err := migrate.CopyStable(ctx, dst, src, nil, nil, stableProgress); err != nil {\n\t\tdst.Close()\n\t\tsrc.Close()\n\t\tcleanupWAL(walDir)\n\t\treturn fmt.Errorf(\"failed to copy stable store: %w\", err)\n\t}\n\n\t// Verify data integrity before finalizing.\n\tsendProgress(progress, \"verifying migrated data\")\n\tif err := verifyMigration(src, dst); err != nil {\n\t\tdst.Close()\n\t\tsrc.Close()\n\t\tcleanupWAL(walDir)\n\t\treturn fmt.Errorf(\"data verification failed: %w\", err)\n\t}\n\n\t// Close both stores before renaming files.\n\tif err := dst.Close(); err != nil {\n\t\tsrc.Close()\n\t\tcleanupWAL(walDir)\n\t\treturn fmt.Errorf(\"failed to close WAL store: %w\", err)\n\t}\n\n\tif err := src.Close(); err != nil {\n\t\tcleanupWAL(walDir)\n\t\treturn fmt.Errorf(\"failed to close BoltDB store: %w\", err)\n\t}\n\n\t// Rename the old BoltDB file to preserve it as a backup with timestamp.\n\ttimestamp := time.Now().Format(\"20060102-150405\")\n\tbackupPath := fmt.Sprintf(\"%s.migrated.%s\", boltPath, timestamp)\n\tif err := os.Rename(boltPath, backupPath); err != nil {","sourceCodeStart":111,"sourceCodeEnd":147,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/raftutil/migrate.go#L111-L147","documentation":"After copying logs and stable data, verifyMigration compares first/last log indexes, key stable-store values, and spot-checks sample log entries between source and destination. This error wraps any verification mismatch or read failure, meaning the migrated WAL copy is not provably identical to the BoltDB source. The migration is aborted and the WAL directory removed, preserving raft.db.","triggerScenarios":"verifyMigration(src, dst) returns an error: first/last index mismatch between stores, CurrentTerm/LastVoteTerm/LastVoteCand mismatch, a sample log entry (first, middle, last) differing in index/term/type/data, or an error reading indexes/logs from either store.","commonSituations":"Concurrent writes to raft.db during migration (the Nomad server was still running); storage silently corrupting data; a bug or interruption that left the WAL copy incomplete despite CopyLogs/CopyStable returning nil.","solutions":["Confirm the Nomad server is fully stopped before migrating — concurrent writes are the leading cause of verification mismatches.","Read the wrapped error to identify which check failed (index vs stable key vs log entry) and compare against raft.db.","Re-run MigrateToWAL from a clean state after stopping all writers.","If mismatches persist on a healthy quiesced cluster, check storage integrity and raft-wal/raft-boltdb versions."],"exampleFix":"// before\nsystemctl start nomad   # server restarted too early\n// after\nsystemctl stop nomad && ./nomad operator raft migrate-wal && systemctl start nomad","handlingStrategy":"validation","validationCode":"// before migrating, guarantee no writers:\n//   systemctl stop nomad (all server nodes)\n//   lsof <raftDir>/raft.db  -> expect no output\nif out, err := exec.Command(\"lsof\", filepath.Join(raftDir, \"raft.db\")).Output(); err == nil && len(out) > 0 {\n    return fmt.Errorf(\"raft.db is still open by another process; stop the server first\")\n}","typeGuard":null,"tryCatchPattern":"err := raftutil.MigrateToWAL(ctx, raftDir, progress)\nif err != nil && strings.Contains(err.Error(), \"data verification failed\") {\n    // do NOT start the server on the wal directory;\n    // raft.db is intact — investigate and retry after quiescing writers\n}","preventionTips":["Never migrate a live server — stop Nomad on the node first.","Check no other process holds raft.db open (lsof) before starting.","Treat verification failure as a hard stop: raft.db is authoritative and untouched.","Use ECC/local storage; silent corruption triggers verification mismatches."],"tags":["migration","data-integrity","verification","raft"],"backgroundTag":"data-verification-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}