{"record":{"id":"4cfb2b89e65307d8","repo":"hashicorp/nomad","slug":"failed-to-read-log-entry-at-index-d-v","errorCode":null,"errorMessage":"failed to read log entry at index %d: %v","messagePattern":"failed to read log entry at index (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"helper/raftutil/fsm.go","lineNumber":138,"sourceCode":"\n\t\tif index != 0 {\n\t\t\tf.nextIdx = index + 1\n\t\t\treturn index, term, nil\n\t\t}\n\t}\n\n\tif f.nextIdx < f.logFirstIdx {\n\t\treturn 0, 0, fmt.Errorf(\"missing logs [%v, %v]\", f.nextIdx, f.logFirstIdx-1)\n\t}\n\n\tif f.nextIdx > f.logLastIdx {\n\t\treturn 0, 0, ErrNoMoreLogs\n\t}\n\n\tvar e raft.Log\n\terr = f.store.GetLog(f.nextIdx, &e)\n\tif err != nil {\n\t\treturn 0, 0, fmt.Errorf(\"failed to read log entry at index %d: %v\", f.nextIdx, err)\n\t}\n\n\tdefer func() {\n\t\tr := recover()\n\t\tif r != nil && strings.HasPrefix(fmt.Sprint(r), \"failed to apply request\") {\n\t\t\t// Enterprise specific log entries will fail to load in OSS repository with \"failed to apply request.\"\n\t\t\t// If not relevant to investigation, we can ignore them and simply worn.\n\t\t\tf.logger.Warn(\"failed to apply log; loading Enterprise data-dir in OSS binary?\", \"index\", e.Index)\n\n\t\t\tf.nextIdx++\n\t\t} else if r != nil {\n\t\t\tpanic(r)\n\t\t}\n\t}()\n\n\tif e.Type == raft.LogCommand {\n\t\tf.fsm.Apply(&e)\n\t}","sourceCodeStart":120,"sourceCodeEnd":156,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/raftutil/fsm.go#L120-L156","documentation":"ApplyNext wraps the error from store.GetLog when the Raft log entry at nextIdx cannot be read from the underlying log store (BoltDB/WAL). This indicates backend read failure at a specific index — distinct from compaction gaps or end-of-log.","triggerScenarios":"Calling ApplyNext when the log store returns an error for GetLog at the current nextIdx: corrupt BoltDB page, WAL read error, I/O failure, or entry missing despite passing the range checks.","commonSituations":"Corrupted raft.db after a crash or power loss; failing disk sectors; WAL file truncation; reading a data dir on damaged storage.","solutions":["Inspect the wrapped error and run integrity checks on the storage backend (bolt check for BoltDB)","Restore the data dir from the latest Raft snapshot or backup","Check disk health (smartctl/dmesg) and filesystem errors if corruption is indicated"],"exampleFix":"// before\nidx, term, err := f.ApplyNext()\nif err != nil { log.Fatal(err) }\n// after\nidx, term, err := f.ApplyNext()\nif err != nil && strings.Contains(err.Error(), \"failed to read log entry\") {\n    log.Printf(\"log store corrupt at index %d: %v — restoring from snapshot\", nextIdx, err)\n    return restoreFromSnapshot(dataDir)\n}","handlingStrategy":"fallback","validationCode":"if err := boltIntegrityCheck(filepath.Join(dataDir, \"raft.db\")); err != nil {\n    return fmt.Errorf(\"raft.db corrupt — restore from snapshot before replay: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"if _, _, err := f.ApplyNext(); err != nil && strings.Contains(err.Error(), \"failed to read log entry\") {\n    log.Printf(\"log store read failure: %v — falling back to snapshot restore\", err)\n    return restoreFromSnapshot(dataDir)\n}","preventionTips":["Take regular Raft snapshots so a corrupt log tail can be discarded","Monitor disk health and filesystem errors on servers","Shut down cleanly (avoid power loss) to reduce BoltDB corruption risk","Verify data dirs with bolt check after incidents before replaying"],"tags":["raft","corruption","storage","go"],"backgroundTag":"raft-log-read-failure","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}