{"record":{"id":"5c5035259ada2889","repo":"hashicorp/nomad","slug":"stable-key-s-mismatch-source-d-destination-d","errorCode":null,"errorMessage":"stable key %s mismatch: source=%d, destination=%d","messagePattern":"stable key (.+?) mismatch: source=(.+?), destination=(.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"helper/raftutil/migrate.go","lineNumber":294,"sourceCode":"\tif srcLast != dstLast {\n\t\treturn fmt.Errorf(\"last index mismatch: source=%d, destination=%d\", srcLast, dstLast)\n\t}\n\n\t// Verify stable store uint64 keys.\n\tuint64Keys := []string{\"CurrentTerm\", \"LastVoteTerm\"}\n\tfor _, key := range uint64Keys {\n\t\tsrcVal, err := src.GetUint64([]byte(key))\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to get source uint64 key %s: %w\", key, err)\n\t\t}\n\n\t\tdstVal, err := dst.GetUint64([]byte(key))\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to get destination uint64 key %s: %w\", key, err)\n\t\t}\n\n\t\tif srcVal != dstVal {\n\t\t\treturn fmt.Errorf(\"stable key %s mismatch: source=%d, destination=%d\", key, srcVal, dstVal)\n\t\t}\n\t}\n\n\t// Verify stable store byte keys.\n\tbyteKeys := []string{\"LastVoteCand\"}\n\tfor _, key := range byteKeys {\n\t\tsrcVal, err := src.Get([]byte(key))\n\t\tif err != nil {\n\t\t\tsrcVal = nil\n\t\t}\n\n\t\tdstVal, err := dst.Get([]byte(key))\n\t\tif err != nil {\n\t\t\tdstVal = nil\n\t\t}\n\n\t\tif string(srcVal) != string(dstVal) {\n\t\t\treturn fmt.Errorf(\"stable key %s mismatch: source=%q, destination=%q\", key, srcVal, dstVal)","sourceCodeStart":276,"sourceCodeEnd":312,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/raftutil/migrate.go#L276-L312","documentation":"verifyMigration compares CurrentTerm and LastVoteTerm between source and destination stable stores after a migration; this error (migrate.go:294) means the uint64 values differ. The migration is aborted because divergent term/vote state can cause Raft voting invariant violations or lost leadership state. The error names the key and both observed values.","triggerScenarios":"After MigrateToWAL, dst.GetUint64(key) differs from src.GetUint64(key) for \"CurrentTerm\" or \"LastVoteTerm\" — the copy wrote a stale/zero value, the destination retained data from a previous use, or the node kept running and the term advanced between copy and verify.","commonSituations":"Migrating into a reused bolt DB holding state from an earlier node/cluster; running migration while the Raft node is live; a copy-loop bug skipping uint64 stable keys.","solutions":["Fully stop the Raft node before migrating so term cannot change mid-copy.","Delete or empty the destination store so stale CurrentTerm/LastVoteTerm cannot survive.","Check that MigrateToWAL's copy loop persists both uint64 keys before verification.","Use the printed source/destination values to determine whether the destination is stale or the copy dropped the key.","Re-run the migration and confirm verifyMigration returns nil."],"exampleFix":"// before\nraftutil.MigrateToWAL(src, existingDstPath) // dst holds stale CurrentTerm from old node\n// after\nraft.Shutdown() // halt raft so term cannot change\nos.Remove(existingDstPath)   // remove stale destination\nraftutil.MigrateToWAL(src, existingDstPath)","handlingStrategy":"validation","validationCode":"// Before migrating, ensure the destination has no stale stable state:\nfor _, key := range []string{\"CurrentTerm\", \"LastVoteTerm\"} {\n    if _, err := dst.GetUint64([]byte(key)); err == nil {\n        return fmt.Errorf(\"destination already holds %s; use a fresh store\", key)\n    }\n}","typeGuard":null,"tryCatchPattern":"if err := raftutil.MigrateToWAL(src, dstPath); err != nil {\n    if strings.Contains(err.Error(), \"stable key\") && strings.Contains(err.Error(), \"mismatch\") {\n        // destination is stale or copy dropped a key; wipe and retry\n        os.Remove(dstPath)\n    }\n    return err\n}","preventionTips":["Never migrate into a store previously used by another node/cluster.","Shut down Raft before migrating so CurrentTerm/LastVoteTerm cannot change mid-copy.","Run migration into a freshly created bolt file every time.","Verify both stores read identical values with a dry-run comparison first.","Check MigrateToWAL copies all uint64 stable keys, not just log entries."],"tags":["raft","migration","stable-store","data-integrity"],"backgroundTag":"raft-stable-store-mismatch","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}