apache/hadoop · error · IOException

invalid number of blocks: {}. The maximum number of blocks

Error message

invalid number of blocks: {}.  The maximum number of blocks per file is {}

What it means

readBlocks() deserializes the block list of a block-carrying edit record (e.g. OP_ADD): it reads a 4-byte block count, then that many Block structs. The count must be between 0 and MAX_BLOCKS = 67,108,864 (1024*1024*64); anything larger is rejected so the NameNode never tries to allocate a multi-gigabyte array from a garbage count. A value above the cap means the stream is corrupt or misaligned, not that a real file had too many blocks.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java:696

        }
        // read clientId and callId
        readRpcIds(in, logVersion);
      } else {
        this.clientName = "";
        this.clientMachine = "";
      }
    }

    static final public int MAX_BLOCKS = 1024 * 1024 * 64;
    
    private static Block[] readBlocks(
        DataInputStream in,
        int logVersion) throws IOException {
      int numBlocks = in.readInt();
      if (numBlocks < 0) {
        throw new IOException("invalid negative number of blocks");
      } else if (numBlocks > MAX_BLOCKS) {
        throw new IOException("invalid number of blocks: " + numBlocks +
            ".  The maximum number of blocks per file is " + MAX_BLOCKS);
      }
      Block[] blocks = new Block[numBlocks];
      for (int i = 0; i < numBlocks; i++) {
        Block blk = new Block();
        blk.readFields(in);
        blocks[i] = blk;
      }
      return blocks;
    }

    public String stringifyMembers() {
      StringBuilder builder = new StringBuilder();
      builder.append("[length=")
          .append(length)
          .append(", inodeId=")
          .append(inodeId)
          .append(", path=")

View on GitHub (pinned to 2add963021)

Solutions

1. Run 'hdfs namenode -recover' and answer yes to skip broken edits, then verify the namespace with 'hdfs fsck /' and immediately write a fresh checkpoint
2. Locate the corrupt transaction first: 'hdfs offlineEditsViewer -i <edits segment> -o /tmp/edits.xml' (it fails exactly at the bad record and prints its txid)
3. Replace the corrupt segment with a healthy copy from the QJM majority / SecondaryNameNode, or fall back to the last good fsimage plus valid edits
4. Test the journal and name disks (smartctl, fsck) and memtest the node before trusting them again

Example fix

# before
hdfs namenode   # IOException: invalid number of blocks: 1893234546. The maximum number of blocks per file is 67108864
# after
hdfs namenode -recover   # skip the damaged edits, then: hdfs fsck / && hdfs dfsadmin -saveNamespace

# validate every segment before NameNode start / before replay
for f in /dfs/name/current/edits_*; do
  hdfs offlineEditsViewer -i "$f" -o /dev/null || echo "CORRUPT: $f"
done

try {
  editLogInputStream.readOp(); // or fsImage.loadEdits(...) at startup
} catch (IOException ioe) {
  LOG.error("Corrupt edit record in " + segment, ioe);
  // re-run NameNode with -recover (skipBrokenEdits=true) to skip past it
}

When it happens

Trigger: NameNode startup replay (or offlineEditsViewer / JournalNode tailing / bootstrapStandby) parses an edits segment where an earlier record was mis-parsed so the stream now sits on a body byte, or the count int itself is corrupt (torn write, bit rot). Also triggered when a segment written by an incompatible layout version is parsed with the wrong version header.

Common situations: NameNode restart after a crash or full journal disk left a partially written record; failing name.dir or journal disk; a stray non-edits file picked up from the current directory; rare downgrade/mixed-version storage dirs.

Related errors

• Mismatched block IDs or generation stamps for the old last b
• Mismatched block IDs or generation stamps, attempting to rep
• Trying to remove more than one block from file {}
• Trying to delete non-existant block {}
• invalid negative number of blocks

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/03bd7f4c49f1acc5. Report an issue: GitHub.