apache/hadoop · error · java.io.IOException

got unexpected exception {message}

Error message

got unexpected exception {message}

What it means

Reader.readOp(skipBrokenEdits=false) catches Throwable from decodeOp(): FSEditLogOp#decodeOp is contractually IOException-only, so any other failure (NegativeArraySizeException, OutOfMemoryError from garbage-driven allocation, protobuf RuntimeExceptions) is wrapped as 'got unexpected exception <msg>' with the original as cause. With skipBrokenEdits=true (recovery mode) the same failure just advances one byte and retries the decode.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java:5084

        try {
          return decodeOp();
        } catch (IOException e) {
          in.reset();
          if (!skipBrokenEdits) {
            throw e;
          }
        } catch (RuntimeException e) {
          // FSEditLogOp#decodeOp is not supposed to throw RuntimeException.
          // However, we handle it here for recovery mode, just to be more
          // robust.
          in.reset();
          if (!skipBrokenEdits) {
            throw e;
          }
        } catch (Throwable e) {
          in.reset();
          if (!skipBrokenEdits) {
            throw new IOException("got unexpected exception " +
                e.getMessage(), e);
          }
        }
        // Move ahead one byte and re-try the decode process.
        if (in.skip(1) < 1) {
          return null;
        }
      }
    }

    void verifyTerminator() throws IOException {
      /** The end of the edit log should contain only 0x00 or 0xff bytes.
       * If it contains other bytes, the log itself may be corrupt.
       * It is important to check this; if we don't, a stray OP_INVALID byte 
       * could make us stop reading the edit log halfway through, and we'd never
       * know that we had lost data.
       */
      limiter.clearLimit();

View on GitHub (pinned to 2add963021)

Solutions

1. Read the wrapped cause in the stack trace first — it identifies the real failure and the failing op
2. Run 'hdfs namenode -recover' (the skipBrokenEdits path) to get past the record byte-by-byte
3. Locate the bad txid with 'hdfs offlineEditsViewer' and recover from the last checkpoint or healthy journal copies
4. If the same segment decodes fine on another node or release, suspect version skew — replay it with the matching release and checkpoint

hdfs offlineEditsViewer -i <edits segment> -o /dev/null   # surfaces the underlying decode failure pre-start

try {
  reader.readOp(false); // skipBrokenEdits=false
} catch (IOException e) {
  Throwable cause = e.getCause(); // the 'unexpected exception' is always wrapped
  LOG.error("decode failed on " + segment, cause);
  // classify via cause, then re-run with skipBrokenEdits=true (namenode -recover)
}

When it happens

Trigger: Any edit-log corruption that makes decoding fail with a non-IOException during normal NameNode startup replay, JournalNode tailing, or checkpointing on a SecondaryNameNode; occasionally a genuine decoder bug such as an op type unknown to the running binary.

Common situations: Same root causes as the specific parse errors (torn writes, bit rot, version skew); the nested cause in the stack trace names the actual failure and usually maps to one of the more specific edit-log errors.

Related errors

• The log file {} seems to contain valid transactions ; journa
• Same delegation token being added twice; invalid entry in fs
• No header found in log
• EOF while reading layout flags from log
• Reached EOF when reading log header

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/65e0472ff4da68a5. Report an issue: GitHub.