{"record":{"id":"4fcde260209dcb71","repo":"hashicorp/nomad","slug":"log-entry-d-mismatch","errorCode":null,"errorMessage":"log entry %d mismatch","messagePattern":"log entry (.+?) mismatch","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"helper/raftutil/migrate.go","lineNumber":338,"sourceCode":"\t\t\t// Check middle entry.\n\t\t\tmiddle := srcFirst + (srcLast-srcFirst)/2\n\t\t\tindicesToCheck = append(indicesToCheck, middle)\n\t\t\t// Check last entry.\n\t\t\tindicesToCheck = append(indicesToCheck, srcLast)\n\t\t}\n\n\t\tfor _, idx := range indicesToCheck {\n\t\t\tvar srcLog, dstLog raft.Log\n\t\t\tif err := src.GetLog(idx, &srcLog); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to get source log %d: %w\", idx, err)\n\t\t\t}\n\t\t\tif err := dst.GetLog(idx, &dstLog); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to get destination log %d: %w\", idx, err)\n\t\t\t}\n\n\t\t\tif srcLog.Index != dstLog.Index || srcLog.Term != dstLog.Term ||\n\t\t\t\tsrcLog.Type != dstLog.Type || string(srcLog.Data) != string(dstLog.Data) {\n\t\t\t\treturn fmt.Errorf(\"log entry %d mismatch\", idx)\n\t\t\t}\n\t\t}\n\t}\n\n\treturn nil\n}\n","sourceCodeStart":320,"sourceCodeEnd":345,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/raftutil/migrate.go#L320-L345","documentation":"verifyMigration spot-checks log entries (first, middle, last) after a Raft log migration to confirm the destination store holds identical entries to the source. It compares Index, Term, Type, and raw Data bytes; any difference returns \"log entry %d mismatch\". This is a data-integrity guard: the migrated log at that index is corrupt, truncated, or was written by an incompatible path.","triggerScenarios":"MigrateToWAL (or the test TestVerifyMigration_IndexMismatch) calls verifyMigration after copying a Raft log, and the destination log at the sampled index has a different Term, Type, Index, or Data payload than the source log at the same index.","commonSituations":"A partially-completed copy (destination truncated mid-migration); logs appended to the destination after the copy so indices shifted; an off-by-one or skipped-entry bug in the migration writer; restoring onto a store that already contained entries at those indices.","solutions":["Delete the partially-migrated destination log store and re-run the migration from a clean, empty destination.","Dump both logs at the failing index (e.g. with raft-tools or GetLog) and compare Term/Type/Data to find whether entries shifted or were skipped during the copy.","Ensure the source store is not receiving new Raft appends during migration (stop the server or take a consistent snapshot first).","If the destination had pre-existing entries, wipe it before migrating so index 1..N map 1:1 with the source."],"exampleFix":"// before: migrating into a destination that already had logs, causing index shift\nerr := raftutil.MigrateToWAL(boltStore, walStore) // \"log entry 5 mismatch\"\n// after: start from an empty destination\nos.RemoveAll(dstWALDir)\nwal, _ := raftwal.NewWAL(...)\nerr := raftutil.MigrateToWAL(boltStore, wal) // clean 1:1 copy, verification passes","handlingStrategy":"validation","validationCode":"// Before migrating, confirm the destination is empty so indices map 1:1\nfirst, last, err := dstStore.FirstIndex()/LastIndex()\nif first != 0 || last != 0 {\n    return fmt.Errorf(\"destination not empty (first=%d last=%d); wipe it before migration\", first, last)\n}","typeGuard":null,"tryCatchPattern":"if err := raftutil.MigrateToWAL(src, dst); err != nil {\n    var mismatchErr interface{ Error() string }\n    if strings.Contains(err.Error(), \"log entry \") && strings.Contains(err.Error(), \"mismatch\") {\n        // integrity failure: do NOT trust the destination; wipe and re-migrate\n        wipeDestination(dst)\n        return retryMigration(src, dst)\n    }\n    return err\n}","preventionTips":["Always migrate into a freshly-initialized, empty log store.","Quiesce the source cluster (stop Raft appends) before copying logs.","Run the migration against a copy of the source so a failed verify leaves production untouched.","Keep the source snapshot as a recovery point until MigrateToWAL completes verification."],"tags":["raft","data-integrity","migration"],"backgroundTag":"raft-log-migration-mismatch","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}