{"record":{"id":"f02ba3938e02066c","repo":"apache/hadoop","slug":"error-replaying-edit-log-at-offset-expected-t","errorCode":null,"errorMessage":"Error replaying edit log at offset {}.  Expected transaction ID was {}","messagePattern":"Error replaying edit log at offset (.+?)\\.  Expected transaction ID was (.+?)","errorType":"exception","errorClass":"EditLogInputException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java","lineNumber":258,"sourceCode":"    try {\n      while (true) {\n        try {\n          FSEditLogOp op;\n          try {\n            op = in.readOp();\n            if (op == null) {\n              break;\n            }\n          } catch (Throwable e) {\n            // Handle a problem with our input\n            check203UpgradeFailure(in.getVersion(true), e);\n            String errorMessage =\n              formatEditLogReplayError(in, recentOpcodeOffsets, expectedTxId);\n            FSImage.LOG.error(errorMessage, e);\n            if (recovery == null) {\n               // We will only try to skip over problematic opcodes when in\n               // recovery mode.\n              throw new EditLogInputException(errorMessage, e, numEdits);\n            }\n            MetaRecoveryContext.editLogLoaderPrompt(\n                \"We failed to read txId \" + expectedTxId,\n                recovery, \"skipping the bad section in the log\");\n            in.resync();\n            continue;\n          }\n          recentOpcodeOffsets[(int)(numEdits % recentOpcodeOffsets.length)] =\n            in.getPosition();\n          if (op.hasTransactionId()) {\n            if (op.getTransactionId() > expectedTxId) { \n              MetaRecoveryContext.editLogLoaderPrompt(\"There appears \" +\n                  \"to be a gap in the edit log.  We expected txid \" +\n                  expectedTxId + \", but got txid \" +\n                  op.getTransactionId() + \".\", recovery, \"ignoring missing \" +\n                  \" transaction IDs\");\n            } else if (op.getTransactionId() < expectedTxId) { \n              MetaRecoveryContext.editLogLoaderPrompt(\"There appears \" +","sourceCodeStart":240,"sourceCodeEnd":276,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java#L240-L276","documentation":"During edit log replay (NameNode startup or standby tailing), an operation could not be read or decoded at the given stream offset while the loader expected transaction id X in strict sequence. FSEditLogLoader formats the message with recent opcode offsets, logs it, and when recovery mode is off rethrows it as EditLogInputException. Causes: corrupt, truncated, or gapped edit logs. With startup option RECOVER the loader instead prompts and skips the bad section via in.resync().","triggerScenarios":"NameNode crash leaving a torn final edit segment; corrupted segment bytes on disk, NFS, or a JournalNode; segments that overlap or leave a txid gap relative to the loaded fsimage; replaying a segment set that does not line up with the image checkpoint.","commonSituations":"Power loss or kernel panic on the NameNode host; disk or NFS corruption; standby NameNode resuming with stale or partially copied edits; fsimage restored from one backup and edits from another.","solutions":["Start the NameNode in recovery mode and skip the bad section: 'hdfs namenode -recover'","With QJM, check every JournalNode for intact segments; repair or remove the bad segment so a valid majority remains, then restart the NameNode","Restore a consistent fsimage plus edits pair from one backup (same checkpoint epoch) instead of mixing files","For a standby, re-bootstrap from the active NameNode with 'hdfs namenode -bootstrapStandby' rather than copying files by hand"],"exampleFix":"# before: normal start fails with EditLogInputException\nhdfs --daemon start namenode\n\n# after: guided recovery, skip the corrupt section\nhdfs namenode -recover\n# at the prompt choose to skip the bad section, then let the NameNode save a fresh checkpoint","handlingStrategy":"try-catch","validationCode":"# before promoting or restarting: prove the segment parses end to end\nhdfs oev -i edits_inprogress_0000000000000001234 -o /tmp/check.xml -p xml\n# parse failure means a corrupt tail: run recovery before relying on the segment","typeGuard":"public static long loadedOps(IOException e) {\n  if (e instanceof EditLogInputException) {\n    return ((EditLogInputException) e).getNumEditsLoaded(); // ops applied before failure\n  }\n  return -1;\n}","tryCatchPattern":"try {\n  loader.loadFSEdits(storage, 0);\n} catch (EditLogInputException elie) {\n  LOG.warn(\"Loaded \" + elie.getNumEditsLoaded()\n      + \" ops before failure: \" + elie.getMessage());\n  // decide: abort, or restart the NameNode with -recover to skip the bad section\n  // (this is the pattern EditLogTailer uses for standby tailing)\n}","preventionTips":["Run 3 JournalNodes on separate hosts so no single bad copy blocks recovery","Take a checkpoint before planned maintenance so the replay window is small","Never restore fsimage and edits from different backups","Monitor NameNode and JournalNode disks with SMART checks"],"tags":["hdfs","namenode","edit-log","corruption","recovery-mode","txid","startup"],"backgroundTag":"edit-log-corruption","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}