{"record":{"id":"90ae859fdbe3b5c1","repo":"hashicorp/nomad","slug":"failed-to-get-destination-uint64-key-s-w","errorCode":null,"errorMessage":"failed to get destination uint64 key %s: %w","messagePattern":"failed to get destination uint64 key (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"helper/raftutil/migrate.go","lineNumber":290,"sourceCode":"\tif srcFirst != dstFirst {\n\t\treturn fmt.Errorf(\"first index mismatch: source=%d, destination=%d\", srcFirst, dstFirst)\n\t}\n\n\tif srcLast != dstLast {\n\t\treturn fmt.Errorf(\"last index mismatch: source=%d, destination=%d\", srcLast, dstLast)\n\t}\n\n\t// Verify stable store uint64 keys.\n\tuint64Keys := []string{\"CurrentTerm\", \"LastVoteTerm\"}\n\tfor _, key := range uint64Keys {\n\t\tsrcVal, err := src.GetUint64([]byte(key))\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to get source uint64 key %s: %w\", key, err)\n\t\t}\n\n\t\tdstVal, err := dst.GetUint64([]byte(key))\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to get destination uint64 key %s: %w\", key, err)\n\t\t}\n\n\t\tif srcVal != dstVal {\n\t\t\treturn fmt.Errorf(\"stable key %s mismatch: source=%d, destination=%d\", key, srcVal, dstVal)\n\t\t}\n\t}\n\n\t// Verify stable store byte keys.\n\tbyteKeys := []string{\"LastVoteCand\"}\n\tfor _, key := range byteKeys {\n\t\tsrcVal, err := src.Get([]byte(key))\n\t\tif err != nil {\n\t\t\tsrcVal = nil\n\t\t}\n\n\t\tdstVal, err := dst.Get([]byte(key))\n\t\tif err != nil {\n\t\t\tdstVal = nil","sourceCodeStart":272,"sourceCodeEnd":308,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/raftutil/migrate.go#L272-L308","documentation":"This error is returned by verifyMigration in helper/raftutil/migrate.go:290 when reading a uint64 stable-store key (CurrentTerm or LastVoteTerm) from the DESTINATION store during a Raft store migration fails. The migration is aborted because the migrated store cannot read keys the source could read, so data integrity cannot be verified. The underlying store error is preserved via %w for diagnosis.","triggerScenarios":"dst.GetUint64([]byte(key)) returns non-nil for \"CurrentTerm\" or \"LastVoteTerm\" while verifyMigration runs after MigrateToWAL — e.g. the copy loop never wrote the key, the destination bolt DB is corrupt, read-only, or not properly committed.","commonSituations":"Migrating into a destination file with wrong permissions or on a full disk; destination store from an older version lacking those keys; partially written destination from a previously failed migration; unit tests (TestVerifyMigration_IndexMismatch) with partially populated destinations.","solutions":["Inspect the wrapped error (errors.Unwrap) to find the real store failure (permissions, disk full, corruption).","Re-run the migration against a fresh, empty destination path so all stable keys get written before verification.","Verify MigrateToWAL's copy loop actually persists CurrentTerm and LastVoteTerm before calling verifyMigration.","Confirm the destination store implementation supports GetUint64 and is opened read-write.","Fix filesystem issues: permissions on the destination DB file and free disk space."],"exampleFix":"// before\nraftutil.MigrateToWAL(src, dstPath) // destination partially written, verify fails\n// after\nif err := os.RemoveAll(dstPath); err != nil { return err } // start clean\nif err := raftutil.MigrateToWAL(src, dstPath); err != nil { return err }","handlingStrategy":"validation","validationCode":"for _, key := range []string{\"CurrentTerm\", \"LastVoteTerm\"} {\n    if _, err := dst.GetUint64([]byte(key)); err != nil {\n        return fmt.Errorf(\"destination unreadable for %s before migration: %w\", key, err)\n    }\n}","typeGuard":null,"tryCatchPattern":"if err := raftutil.MigrateToWAL(src, dstPath); err != nil {\n    log.Printf(\"migration failed: %v (cause: %v)\", err, errors.Unwrap(err))\n    return err\n}","preventionTips":["Always migrate to a fresh, empty destination path.","Check destination file permissions and free disk space before migrating.","Stop the Raft node before migration so state is quiescent.","Inspect the wrapped %w cause immediately; it names the real store failure.","Write both uint64 stable keys to the destination and read them back as a smoke test before verification."],"tags":["raft","migration","storage","data-integrity"],"backgroundTag":"raft-migration-verification-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}