apache/hadoop · critical · java.io.IOException

Read extra bytes after the terminator!

Error message

Read extra bytes after the terminator!

What it means

A finalized edit segment ends with an OP_INVALID opcode followed only by 0x00/0xFF padding. After decodeOp sees OP_INVALID, verifyTerminator() scans every remaining byte and throws if anything other than 0x00 or 0xFF appears. The check exists because a stray OP_INVALID byte mid-log would otherwise silently truncate replay and lose committed transactions.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java:5114

      /** The end of the edit log should contain only 0x00 or 0xff bytes.
       * If it contains other bytes, the log itself may be corrupt.
       * It is important to check this; if we don't, a stray OP_INVALID byte 
       * could make us stop reading the edit log halfway through, and we'd never
       * know that we had lost data.
       */
      limiter.clearLimit();
      int numRead = -1, idx = 0;
      while (true) {
        try {
          numRead = -1;
          idx = 0;
          numRead = in.read(temp);
          if (numRead == -1) {
            return;
          }
          while (idx < numRead) {
            if ((temp[idx] != (byte)0) && (temp[idx] != (byte)-1)) {
              throw new IOException("Read extra bytes after " +
                "the terminator!");
            }
            idx++;
          }
        } finally {
          // After reading each group of bytes, we reposition the mark one
          // byte before the next group.  Similarly, if there is an error, we
          // want to reposition the mark one byte before the error
          if (numRead != -1) { 
            in.reset();
            IOUtils.skipFully(in, idx);
            in.mark(temp.length + 1);
            IOUtils.skipFully(in, 1);
          }
        }
      }
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Treat this as critical corruption: run 'hdfs namenode -recover' only after confirming which transactions follow the terminator with 'hdfs offlineEditsViewer'
2. Restore an intact copy of the segment from the QJM majority / SecondaryNameNode instead of skipping, because skipping here risks losing committed transactions
3. If no clean copy exists, roll back to the previous fsimage checkpoint and re-accept the lost transactions
4. Fix the finalize-path root cause (single journal dir, crash loops, shared-edits misconfiguration) before reusing the journals

hdfs offlineEditsViewer -i <finalized edits segment> -o /dev/null   # replaying to EOF exercises the terminator path

try {
  reader.scanOp(); // or readOp; OP_INVALID triggers the terminator scan
} catch (IOException e) {
  if (e.getMessage().contains("after the terminator")) {
    // CRITICAL: data follows end-of-log marker -> transactions at risk
    // prefer restoring a clean segment copy over blind recovery-mode skipping
  } else { throw e; }
}

When it happens

Trigger: Corruption flips a mid-record byte to 0xFF where an opcode was expected, so the reader treats it as end-of-log and then finds live record bytes during the terminator scan; or a segment was rewritten/refinalized badly so real data follows the terminator (crash during finalize, partially overwritten file).

Common situations: Restart after a crash during segment finalization; journal files reused/overwritten incorrectly by misconfigured shared edits dirs; underlying disk or replication damage.

Related errors

• The log file {} seems to contain valid transactions ; journa
• No log file to finalize at transaction ID {} ; journal id: {
• Same delegation token being added twice; invalid entry in fs
• No header found in log
• EOF while reading layout flags from log

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7968bc5bd1ce5deb. Report an issue: GitHub.