What causes "Unexpected block size: ${numBytes}" in apache/hadoop?

Verify client and server run compatible Hadoop versions (same major line for hdfs-client wire protocol); align versions and retry.

apache/hadoop · error · IOException

Unexpected block size: ${numBytes}

Error message

Unexpected block size: ${numBytes}

What it means

Thrown by Block.readHelper after deserializing a block (blockId, numBytes, generationStamp) from a DataInput stream when numBytes is negative. Block is the wire representation of an HDFS block, so a negative size means the bytes just read are not a valid Block encoding. This is a data-integrity guard against a corrupt or misaligned stream, not a user-input error.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/Block.java:229

  }

  @Override // Writable
  public void readFields(DataInput in) throws IOException {
    readHelper(in);
  }

  final void writeHelper(DataOutput out) throws IOException {
    out.writeLong(blockId);
    out.writeLong(numBytes);
    out.writeLong(generationStamp);
  }

  final void readHelper(DataInput in) throws IOException {
    this.blockId = in.readLong();
    this.numBytes = in.readLong();
    this.generationStamp = in.readLong();
    if (numBytes < 0) {
      throw new IOException("Unexpected block size: " + numBytes);
    }
  }

  // write only the identifier part of the block
  public void writeId(DataOutput out) throws IOException {
    out.writeLong(blockId);
    out.writeLong(generationStamp);
  }

  // Read only the identifier part of the block
  public void readId(DataInput in) throws IOException {
    this.blockId = in.readLong();
    this.generationStamp = in.readLong();
  }

  /**
   * Compares this Block with the specified Block for order. Returns a negative
   * integer, zero, or a positive integer as this Block is less than, equal to,

View on GitHub (pinned to 2add963021)

Solutions

Verify client and server run compatible Hadoop versions (same major line for hdfs-client wire protocol); align versions and retry.
Check DataNode logs for disk read errors or checksum failures on the blockId reported in the message; evict/reschedule the replica if the media is bad.
Inspect the network path (switch, NIC, offloaded checksums/TOE) between client and DataNode for corruption; disable suspect offloads and retry.
If using a proxy or custom IPC layer, verify it does not splice or reorder bytes in the data-transfer channel.

Defensive patterns

Strategy: try-catch

Try / catch

try {
  block.readFields(in);
} catch (IOException e) {
  if (e.getMessage().contains("Unexpected block size")) {
    // stream corrupt / version mismatch — do not retry the same stream
    throw new DataCorruptionException("bad Block encoding", e);
  }
  throw e;
}

Prevention

Keep client and server Hadoop versions within the documented wire-compatibility window during rolling upgrades.
Monitor DataNode disk and checksum metrics so corrupt replicas are evicted before clients read them.
Avoid custom splicing of the data-transfer socket; if unavoidable, add integrity checks (e.g. TCP checksums are weak — use MACs on sensitive links).

When it happens

Trigger: Deserializing a Block from an IPC reply or data-transfer stream that is corrupt, truncated, or out of frame: e.g. a client/server version mismatch where the two sides serialize Block differently, garbage read after a protocol desync, or a partially-flipped buffer from a failing disk or NIC on the DataNode.

Common situations: Mixed Hadoop versions (rolling upgrade with incompatible wire formats), network-level corruption (bad NIC/cable, TCP proxy mangling frames), reading a block report or edit log segment that is truncated. Rarely, faulty direct-byte-buffer or checksum bugs in a custom DataNode.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8a5c348cf09aa418. Report an issue: GitHub.