apache/hadoop · error · IOException

Invalid block array length: {}

Error message

Invalid block array length: {}

What it means

readCompactBlockArray reads a length-prefixed, delta-encoded block list from a legacy-format fsimage/edits stream. The count is a WritableUtils VInt; a negative value means the bytes do not parse as the expected format. The stream is corrupt or truncated, or a layout-version mismatch makes the reader's field expectations disagree with what was written.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java:506

    WritableUtils.writeVInt(out, blocks.length);
    Block prev = null;
    for (Block b : blocks) {
      long szDelta = b.getNumBytes() -
          (prev != null ? prev.getNumBytes() : 0);
      long gsDelta = b.getGenerationStamp() -
          (prev != null ? prev.getGenerationStamp() : 0);
      out.writeLong(b.getBlockId()); // blockid is random
      WritableUtils.writeVLong(out, szDelta);
      WritableUtils.writeVLong(out, gsDelta);
      prev = b;
    }
  }
  
  public static Block[] readCompactBlockArray(
      DataInput in, int logVersion) throws IOException {
    int num = WritableUtils.readVInt(in);
    if (num < 0) {
      throw new IOException("Invalid block array length: " + num);
    }
    Block prev = null;
    Block[] ret = new Block[num];
    for (int i = 0; i < num; i++) {
      long id = in.readLong();
      long sz = WritableUtils.readVLong(in) +
          ((prev != null) ? prev.getNumBytes() : 0);
      long gs = WritableUtils.readVLong(in) +
          ((prev != null) ? prev.getGenerationStamp() : 0);
      ret[i] = new Block(id, sz, gs);
      prev = ret[i];
    }
    return ret;
  }

  public static void writeCacheDirectiveInfo(DataOutputStream out,
      CacheDirectiveInfo directive) throws IOException {
    writeLong(directive.getId(), out);

View on GitHub (pinned to 2add963021)

Solutions

1. Verify integrity: compare fsimage against its fsimage.md5; run 'hdfs oev' on the edits to locate the bad record
2. Read the file with the Hadoop version/layout that wrote it
3. Restore the affected file from another storage directory or backup and retry
4. If corruption is confirmed and no copy exists, fall back to the previous checkpoint plus whatever subsequent edits survive

Example fix

# before: corrupt/truncated file fails to load
md5sum -c /dfs/name/current/fsimage.md5   # FAILED
# after: replace with a verified copy from another dir or backup
cp /dfs/name2/current/fsimage* /dfs/name/current/

// verify checksum before loading a legacy image
MessageDigest md = MessageDigest.getInstance("MD5");
try (InputStream in = new FileInputStream(img)) {
  byte[] b = new byte[8192]; for (int n; (n = in.read(b)) != -1;) md.update(b, 0, n);
}
String actual = String.format("%d", new BigInteger(1, md.digest()));
// compare against the hex in the sibling fsimage.md5 file

catch (IOException e) when the message begins with "Invalid block array length" — the bytes are corrupt; do not re-read the same file. Fall back to another storage directory's copy or the previous checkpoint, then replay surviving edits.

When it happens

Trigger: Loading a legacy image or replaying edits where the block-count vint decodes negative: byte-level corruption, truncated file, or reading a stream produced by an incompatible Hadoop layout version.

Common situations: Disk corruption or partial copy of fsimage/edits; replaying edits recovered from a journal with a hole; version skew on legacy-format files.

Related errors

• unknown flags set in ModifyCacheDirectiveInfoOp: {}
• Can't read FileStatusProto with negative size of ${size}
• The length of the feature flag section was negative at {} by
• Same delegation token being added twice; invalid entry in fs
• file VERSION is invalid.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/41ae6b172b770f05. Report an issue: GitHub.