{"record":{"id":"8327ee1825977014","repo":"hashicorp/nomad","slug":"failed-to-read-log-entry-at-index-d-firstidx-d","errorCode":null,"errorMessage":"failed to read log entry at index %d (firstIdx: %d, lastIdx: %d): %v","messagePattern":"failed to read log entry at index (.+?) \\(firstIdx: (.+?), lastIdx: (.+?)\\): (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"helper/raftutil/state.go","lineNumber":121,"sourceCode":"// warnings. If opening the raft state returns an error, both channels\n// will be nil.\nfunc LogEntries(p string) (<-chan interface{}, <-chan error, error) {\n\tstore, firstIdx, lastIdx, err := RaftStateInfo(p)\n\tif err != nil {\n\t\treturn nil, nil, fmt.Errorf(\"failed to open raft logs: %v\", err)\n\t}\n\n\tentries := make(chan interface{})\n\twarnings := make(chan error)\n\n\tgo func() {\n\t\tdefer store.Close()\n\t\tdefer close(entries)\n\t\tfor i := firstIdx; i <= lastIdx; i++ {\n\t\t\tvar e raft.Log\n\t\t\terr := store.GetLog(i, &e)\n\t\t\tif err != nil {\n\t\t\t\twarnings <- fmt.Errorf(\n\t\t\t\t\t\"failed to read log entry at index %d (firstIdx: %d, lastIdx: %d): %v\",\n\t\t\t\t\ti, firstIdx, lastIdx, err)\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tentry, err := decode(&e)\n\t\t\tif err != nil {\n\t\t\t\twarnings <- fmt.Errorf(\n\t\t\t\t\t\"failed to decode log entry at index %d: %v\", i, err)\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tentries <- entry\n\t\t}\n\t}()\n\n\treturn entries, warnings, nil\n}","sourceCodeStart":103,"sourceCodeEnd":139,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/raftutil/state.go#L103-L139","documentation":"Inside LogEntries' goroutine, each index from firstIdx to lastIdx is fetched via store.GetLog. If GetLog returns an error for a specific index, it is sent on the warnings channel (not returned as a fatal error) and iteration continues. This means the store advertised an index range but the entry at that index could not be read back.","triggerScenarios":"A gap or corruption in the middle of the log range: an index present in metadata (First/LastIndex) but whose record is missing or unreadable — e.g. corrupt segment in the WAL, corrupted bolt page, or a bolt bucket inconsistency.","commonSituations":"Disk corruption in the middle of raft.db or a WAL segment; data dir partially restored from backup leaving index gaps; hardware failure (bad blocks); interrupted copy of the store to another machine.","solutions":["Read the warnings channel alongside entries and log each failed index; a few isolated gaps may be tolerable for inspection.","Determine the extent: many consecutive failures means broad corruption — restore raft.db / the wal/ directory from a healthy backup or peer.","Run filesystem/disk health checks on the volume; replace failing hardware before rebuilding.","If the server's own state is affected, retire the server and re-join a fresh one so raft re-replicates logs from the leader.","Keep the store open read-only and stop the agent to avoid read failures racing with compaction/writes."],"exampleFix":"// before\nentries, _, err := raftutil.LogEntries(p)\n// warnings discarded — unreadable indexes silently dropped\n// after\nentriesCh, warningsCh, err := raftutil.LogEntries(p)\nif err != nil {\n\treturn err\n}\nfor w := range warningsCh {\n\tlog.Printf(\"raft log gap: %v\", w) // surface corrupt/missing indexes\n}","handlingStrategy":"fallback","validationCode":"func readRangeWithFallback(p string) ([]*logMessage, []error, error) {\n\tentriesCh, warnCh, err := raftutil.LogEntries(p)\n\tif err != nil {\n\t\treturn nil, nil, err\n\t}\n\tvar entries []*logMessage\n\tvar gaps []error\n\tfor entriesCh != nil || warnCh != nil {\n\t\tselect {\n\t\tcase e, ok := <-entriesCh:\n\t\t\tif !ok { entriesCh = nil; continue }\n\t\t\tentries = append(entries, e.(*logMessage))\n\t\tcase w, ok := <-warnCh:\n\t\t\tif !ok { warnCh = nil; continue }\n\t\t\tgaps = append(gaps, w)\n\t\t}\n\t}\n\tif len(gaps) > len(entries) {\n\t\treturn nil, gaps, fmt.Errorf(\"majority of raft log entries unreadable; restore from backup\")\n\t}\n\treturn entries, gaps, nil\n}","typeGuard":"func isLogReadWarning(w error) bool {\n\treturn w != nil && strings.Contains(w.Error(), \"failed to read log entry at index\")\n}","tryCatchPattern":"for w := range warnings {\n\tif isLogReadWarning(w) {\n\t\tvar idx uint64\n\t\tfmt.Sscanf(w.Error(), \"failed to read log entry at index %d\", &idx)\n\t\tlog.Printf(\"gap at index %d — continuing\", idx)\n\t}\n}","preventionTips":["Always drain the warnings channel or you will deadlock the producer goroutine.","Compare the count of warnings vs entries to gauge corruption severity.","Restore from a peer/backup when consecutive index gaps appear.","Stop the agent before reading to avoid reads racing with writes.","Run disk health checks; mid-log corruption often indicates failing hardware."],"tags":["raft","corruption","data-loss","storage"],"backgroundTag":"raft-log-entry-unreadable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}