apache/hadoop · critical · IOException

Invalid packet: data length in packet header exceeds data le

Error message

Invalid packet: data length in packet header exceeds data length received. dataPlusChecksumLen=${dataPlusChecksumLen} header: ${header}

What it means

After parsing the packet header, PacketReceiver computes checksumLen = dataPlusChecksumLen - header.getDataLen(); a negative result means the header claims more data bytes than the packet's payload length can hold — an internally inconsistent header, i.e. corruption or desynchronization. The message dumps both the payload length and full header for diagnosis.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PacketReceiver.java:194

    curPacketBuf.position(PacketHeader.PKT_LENGTHS_LEN);
    curPacketBuf.limit(PacketHeader.PKT_LENGTHS_LEN +
        dataPlusChecksumLen + headerLen);
    doReadFully(ch, in, curPacketBuf);
    curPacketBuf.flip();
    curPacketBuf.position(PacketHeader.PKT_LENGTHS_LEN);

    // Extract the header from the front of the buffer (after the length prefixes)
    byte[] headerBuf = new byte[headerLen];
    curPacketBuf.get(headerBuf);
    if (curHeader == null) {
      curHeader = new PacketHeader();
    }
    curHeader.setFieldsFromData(payloadLen, headerBuf);

    // Compute the sub-slices of the packet
    int checksumLen = dataPlusChecksumLen - curHeader.getDataLen();
    if (checksumLen < 0) {
      throw new IOException("Invalid packet: data length in packet header " +
          "exceeds data length received. dataPlusChecksumLen=" +
          dataPlusChecksumLen + " header: " + curHeader);
    }

    reslicePacket(headerLen, checksumLen, curHeader.getDataLen());
  }

  /**
   * Rewrite the last-read packet on the wire to the given output stream.
   */
  public void mirrorPacketTo(DataOutputStream mirrorOut) throws IOException {
    Preconditions.checkState(!useDirectBuffers,
        "Currently only supported for non-direct buffers");
    mirrorOut.write(curPacketBuf.array(),
        curPacketBuf.arrayOffset(),
        curPacketBuf.remaining());
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the operation so a different replica/pipeline is used; DFSClient-level retries usually mask transient cases.
2. Run 'hdfs fsck -blockId' on the reported block and check sender DataNode logs for disk errors.
3. Ensure uniform Hadoop versions across the pipeline.
4. If persistent, capture traffic and compare announced lengths vs actual bytes to locate the corrupting hop.

try {
  receiver.readNextPacket();
} catch (IOException e) {
  if (e.getMessage().contains("data length in packet header exceeds")) {
    // inconsistent header vs payload — drop the replica/reader and retry
    throw new StreamCorruptException(e.getMessage(), e);
  }
  throw e;
}

When it happens

Trigger: Truncated or corrupted packets where the data length field in PacketHeader exceeds the announced payload: failing disks producing short reads on the sender, TCP stream desync after an earlier framing error, or incompatible PacketHeader layouts between versions.

Common situations: DataNode disks going bad (short/garbage writes into the pipeline), rolling-upgrade version mixes, transport-level corruption, flaky direct-buffer handling in custom builds.

Related errors

• BlockReader: error in packet header {}
• Invalid payload length ${payloadLen}
• Could not create DataChecksum from the byte array of length
• Could not create DataChecksum from the byte array of length
• The value %d does not map to a valid checksum Type

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/193d1ee9031d9b56. Report an issue: GitHub.