{"record":{"id":"8d984bd039ce3366","repo":"hashicorp/nomad","slug":"stable-key-s-mismatch-source-q-destination-q","errorCode":null,"errorMessage":"stable key %s mismatch: source=%q, destination=%q","messagePattern":"stable key (.+?) mismatch: source=%q, destination=%q","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"helper/raftutil/migrate.go","lineNumber":312,"sourceCode":"\t\t\treturn fmt.Errorf(\"stable key %s mismatch: source=%d, destination=%d\", key, srcVal, dstVal)\n\t\t}\n\t}\n\n\t// Verify stable store byte keys.\n\tbyteKeys := []string{\"LastVoteCand\"}\n\tfor _, key := range byteKeys {\n\t\tsrcVal, err := src.Get([]byte(key))\n\t\tif err != nil {\n\t\t\tsrcVal = nil\n\t\t}\n\n\t\tdstVal, err := dst.Get([]byte(key))\n\t\tif err != nil {\n\t\t\tdstVal = nil\n\t\t}\n\n\t\tif string(srcVal) != string(dstVal) {\n\t\t\treturn fmt.Errorf(\"stable key %s mismatch: source=%q, destination=%q\", key, srcVal, dstVal)\n\t\t}\n\t}\n\n\t// If we have logs, spot-check a few entries for data integrity.\n\tif srcFirst > 0 && srcLast > 0 {\n\t\tindicesToCheck := []uint64{srcFirst}\n\t\tif srcLast > srcFirst {\n\t\t\t// Check middle entry.\n\t\t\tmiddle := srcFirst + (srcLast-srcFirst)/2\n\t\t\tindicesToCheck = append(indicesToCheck, middle)\n\t\t\t// Check last entry.\n\t\t\tindicesToCheck = append(indicesToCheck, srcLast)\n\t\t}\n\n\t\tfor _, idx := range indicesToCheck {\n\t\t\tvar srcLog, dstLog raft.Log\n\t\t\tif err := src.GetLog(idx, &srcLog); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to get source log %d: %w\", idx, err)","sourceCodeStart":294,"sourceCodeEnd":330,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/raftutil/migrate.go#L294-L330","documentation":"verifyMigration compares the byte-valued stable key \"LastVoteCand\" between source and destination with %q formatting (migrate.go:312); this error fires when the candidate IDs differ, including nil vs non-nil when one store errors on Get (errors are treated as nil on both sides). The migration aborts because divergent last-vote candidate state can violate Raft voting invariants.","triggerScenarios":"src.Get([]byte(\"LastVoteCand\")) and dst.Get([]byte(\"LastVoteCand\")) return different byte slices after copy — the copy dropped or truncated the key, the destination holds an old value, or one store errored (treated as nil) while the other held a candidate ID.","commonSituations":"Migrating into a non-empty destination store; a destination whose Get error semantics differ for missing keys (ErrKeyNotFound vs nil) making an unset source key appear as a mismatch; partial copy where byte keys were skipped.","solutions":["Ensure the destination store is empty/fresh so no stale LastVoteCand survives.","Check that MigrateToWAL copies the LastVoteCand byte key and preserves absence on both sides.","Match the destination store's Get error semantics to the source's for missing keys.","Stop the Raft node first so no vote is cast between copy and verification.","Inspect the %q values in the error to see if the destination is stale or the copy truncated the value."],"exampleFix":"// before\ndst, _ := raftboltdb.NewBoltStore(oldPath) // contains LastVoteCand from previous run\nraftutil.MigrateToWAL(src, dst)\n// after\ndeleteStaleStableKeys(dst, []string{\"LastVoteCand\"}) // or start with a fresh bolt file\nraftutil.MigrateToWAL(src, dst)","handlingStrategy":"validation","validationCode":"// Detect divergent LastVoteCand semantics before migrating:\nsrcCand, srcErr := src.Get([]byte(\"LastVoteCand\"))\ndstCand, dstErr := dst.Get([]byte(\"LastVoteCand\"))\nif (srcErr == nil) != (dstErr == nil) || string(srcCand) != string(dstCand) {\n    return fmt.Errorf(\"LastVoteCand diverges before migration; clean the destination\")\n}","typeGuard":null,"tryCatchPattern":"if err := raftutil.MigrateToWAL(src, dstPath); err != nil {\n    if strings.Contains(err.Error(), \"LastVoteCand mismatch\") {\n        log.Printf(\"last-vote state diverged: %v\", err)\n        // wipe destination and retry with the node stopped\n    }\n    return err\n}","preventionTips":["Use a destination store whose Get error semantics for missing keys match the source's.","Start every migration from an empty destination so absence is preserved as absence.","Stop the node so no vote is cast between copy and verify.","Confirm byte keys like LastVoteCand are included in the copy loop.","Treat 'store missing key' errors consistently (map ErrKeyNotFound to nil) in custom store implementations."],"tags":["raft","migration","stable-store","last-vote"],"backgroundTag":"raft-stable-store-mismatch","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}