{"record":{"id":"6380310f11fe1bc1","repo":"hashicorp/nomad","slug":"failed-to-get-source-log-d-w","errorCode":null,"errorMessage":"failed to get source log %d: %w","messagePattern":"failed to get source log (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"helper/raftutil/migrate.go","lineNumber":330,"sourceCode":"\t\t\treturn fmt.Errorf(\"stable key %s mismatch: source=%q, destination=%q\", key, srcVal, dstVal)\n\t\t}\n\t}\n\n\t// If we have logs, spot-check a few entries for data integrity.\n\tif srcFirst > 0 && srcLast > 0 {\n\t\tindicesToCheck := []uint64{srcFirst}\n\t\tif srcLast > srcFirst {\n\t\t\t// Check middle entry.\n\t\t\tmiddle := srcFirst + (srcLast-srcFirst)/2\n\t\t\tindicesToCheck = append(indicesToCheck, middle)\n\t\t\t// Check last entry.\n\t\t\tindicesToCheck = append(indicesToCheck, srcLast)\n\t\t}\n\n\t\tfor _, idx := range indicesToCheck {\n\t\t\tvar srcLog, dstLog raft.Log\n\t\t\tif err := src.GetLog(idx, &srcLog); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to get source log %d: %w\", idx, err)\n\t\t\t}\n\t\t\tif err := dst.GetLog(idx, &dstLog); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to get destination log %d: %w\", idx, err)\n\t\t\t}\n\n\t\t\tif srcLog.Index != dstLog.Index || srcLog.Term != dstLog.Term ||\n\t\t\t\tsrcLog.Type != dstLog.Type || string(srcLog.Data) != string(dstLog.Data) {\n\t\t\t\treturn fmt.Errorf(\"log entry %d mismatch\", idx)\n\t\t\t}\n\t\t}\n\t}\n\n\treturn nil\n}\n","sourceCodeStart":312,"sourceCodeEnd":345,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/raftutil/migrate.go#L312-L345","documentation":"During the spot-check phase of verifyMigration (migrate.go:330), reading a log entry at a sampled index (first, middle, or last) from the SOURCE LogStore via GetLog failed. The source is authoritative, so this means the source itself is unreadable or corrupt and the migration cannot be verified; the index and wrapped cause are returned.","triggerScenarios":"src.GetLog(idx, &srcLog) errors for idx in {srcFirst, middle, srcLast} after index-range checks passed — the source's index metadata says the entry exists but it is corrupt/truncated, the source file is damaged, or an IO error occurs during read.","commonSituations":"Corrupted raft.db or log file after a crash or disk failure; a source implementation whose LastIndex reports entries GetLog cannot return (stale index); log compaction removed entries the source still advertises.","solutions":["Inspect the wrapped error to identify the underlying read failure.","Check source store integrity (bolt integrity check, file permissions, disk health).","Reconcile srcFirst/srcLast with what GetLog can actually return; restore from backup if corrupt.","If compaction legitimately removed entries, fix the source's index reporting before migrating.","Retry the migration after repairing or replacing the source store."],"exampleFix":"// before\nraftutil.MigrateToWAL(corruptSrc, dstPath) // srcLast points at unreadable entry\n// after\nif err := boltCheck(srcPath); err != nil {\n    src = restoreFromBackup(srcPath) // repair source first\n}\nraftutil.MigrateToWAL(src, dstPath)","handlingStrategy":"fallback","validationCode":"// Sanity-check the source is fully readable before migrating:\nfirst, _ := src.FirstIndex()\nlast, _ := src.LastIndex()\nfor _, idx := range []uint64{first, first + (last-first)/2, last} {\n    var l raft.Log\n    if err := src.GetLog(idx, &l); err != nil {\n        return fmt.Errorf(\"source unreadable at %d; repair/restore before migrating: %w\", idx, err)\n    }\n}","typeGuard":null,"tryCatchPattern":"if err := raftutil.MigrateToWAL(src, dstPath); err != nil {\n    if strings.Contains(err.Error(), \"failed to get source log\") {\n        // source corrupt: restore from backup instead of retrying\n        src = restoreFromBackup(srcPath)\n    }\n    return err\n}","preventionTips":["Run integrity checks on the source store periodically and before any migration.","Keep verified backups of the source raft.db for restore-on-corruption.","Ensure LastIndex only reports entries GetLog can actually return in custom LogStore implementations.","Monitor disk health (SMART) on nodes holding raft logs.","Account for log compaction: never migrate a store whose index metadata outlives its entries."],"tags":["raft","logstore","corruption","migration"],"backgroundTag":"raft-log-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}