apache/hadoop · critical · EditLogInputException

Error replaying edit log at offset {}. Expected transaction

Error message

Error replaying edit log at offset {}.  Expected transaction ID was {}

What it means

During edit log replay (NameNode startup or standby tailing), an operation could not be read or decoded at the given stream offset while the loader expected transaction id X in strict sequence. FSEditLogLoader formats the message with recent opcode offsets, logs it, and when recovery mode is off rethrows it as EditLogInputException. Causes: corrupt, truncated, or gapped edit logs. With startup option RECOVER the loader instead prompts and skips the bad section via in.resync().

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java:258

    try {
      while (true) {
        try {
          FSEditLogOp op;
          try {
            op = in.readOp();
            if (op == null) {
              break;
            }
          } catch (Throwable e) {
            // Handle a problem with our input
            check203UpgradeFailure(in.getVersion(true), e);
            String errorMessage =
              formatEditLogReplayError(in, recentOpcodeOffsets, expectedTxId);
            FSImage.LOG.error(errorMessage, e);
            if (recovery == null) {
               // We will only try to skip over problematic opcodes when in
               // recovery mode.
              throw new EditLogInputException(errorMessage, e, numEdits);
            }
            MetaRecoveryContext.editLogLoaderPrompt(
                "We failed to read txId " + expectedTxId,
                recovery, "skipping the bad section in the log");
            in.resync();
            continue;
          }
          recentOpcodeOffsets[(int)(numEdits % recentOpcodeOffsets.length)] =
            in.getPosition();
          if (op.hasTransactionId()) {
            if (op.getTransactionId() > expectedTxId) { 
              MetaRecoveryContext.editLogLoaderPrompt("There appears " +
                  "to be a gap in the edit log.  We expected txid " +
                  expectedTxId + ", but got txid " +
                  op.getTransactionId() + ".", recovery, "ignoring missing " +
                  " transaction IDs");
            } else if (op.getTransactionId() < expectedTxId) { 
              MetaRecoveryContext.editLogLoaderPrompt("There appears " +

View on GitHub (pinned to 2add963021)

Solutions

1. Start the NameNode in recovery mode and skip the bad section: 'hdfs namenode -recover'
2. With QJM, check every JournalNode for intact segments; repair or remove the bad segment so a valid majority remains, then restart the NameNode
3. Restore a consistent fsimage plus edits pair from one backup (same checkpoint epoch) instead of mixing files
4. For a standby, re-bootstrap from the active NameNode with 'hdfs namenode -bootstrapStandby' rather than copying files by hand

Example fix

# before: normal start fails with EditLogInputException
hdfs --daemon start namenode

# after: guided recovery, skip the corrupt section
hdfs namenode -recover
# at the prompt choose to skip the bad section, then let the NameNode save a fresh checkpoint

# before promoting or restarting: prove the segment parses end to end
hdfs oev -i edits_inprogress_0000000000000001234 -o /tmp/check.xml -p xml
# parse failure means a corrupt tail: run recovery before relying on the segment

public static long loadedOps(IOException e) {
  if (e instanceof EditLogInputException) {
    return ((EditLogInputException) e).getNumEditsLoaded(); // ops applied before failure
  }
  return -1;
}

try {
  loader.loadFSEdits(storage, 0);
} catch (EditLogInputException elie) {
  LOG.warn("Loaded " + elie.getNumEditsLoaded()
      + " ops before failure: " + elie.getMessage());
  // decide: abort, or restart the NameNode with -recover to skip the bad section
  // (this is the pattern EditLogTailer uses for standby tailing)
}

When it happens

Trigger: NameNode crash leaving a torn final edit segment; corrupted segment bytes on disk, NFS, or a JournalNode; segments that overlap or leave a txid gap relative to the loaded fsimage; replaying a segment set that does not line up with the image checkpoint.

Common situations: Power loss or kernel panic on the NameNode host; disk or NFS corruption; standby NameNode resuming with stale or partially copied edits; fsimage restored from one backup and edits from another.

Related errors

• Same delegation token being added twice; invalid entry in fs
• The layout version {} supports inodeId but gave bogus inodeI
• Invalid operation read {}
• Mismatched block IDs or generation stamps for the old last b
• Mismatched block IDs or generation stamps, attempting to rep

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f02ba3938e02066c. Report an issue: GitHub.