apache/hadoop · error · IOException

Expected empty end-of-read packet! Header: {}

Error message

Expected empty end-of-read packet! Header: {}

What it means

When a client reaches the end of a block, the data-transfer protocol requires the datanode to send one final empty packet flagged lastPacketInBlock. readTrailingEmptyPacket throws when that trailer packet carries data or is not marked as the last packet — the datanode violated the end-of-block handshake.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java:273

        break;
      }

      int skip = (int)Math.min(curDataSlice.remaining(), needToSkip);
      curDataSlice.position(curDataSlice.position() + skip);
      skipped += skip;
    }
    return skipped;
  }

  private void readTrailingEmptyPacket() throws IOException {
    LOG.trace("Reading empty packet at end of read");

    packetReceiver.receiveNextPacket(in);

    PacketHeader trailer = packetReceiver.getHeader();
    if (!trailer.isLastPacketInBlock() ||
        trailer.getDataLen() != 0) {
      throw new IOException("Expected empty end-of-read packet! Header: " +
          trailer);
    }
  }

  protected BlockReaderRemote(String file, long blockId,
                              DataChecksum checksum, boolean verifyChecksum,
                              long startOffset, long firstChunkOffset,
                              long bytesToRead, Peer peer,
                              DatanodeID datanodeID, PeerCache peerCache,
                              int networkDistance) {
    // Path is used only for printing block and file information in debug
    this.peer = peer;
    this.datanodeID = datanodeID;
    this.in = peer.getInputStreamChannel();
    this.checksum = checksum;
    this.verifyChecksum = verifyChecksum;
    this.startOffset = Math.max( startOffset, 0 );
    this.filename = file;

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the read; transient cases resolve on a new connection or replica.
2. Confirm client and datanode Hadoop versions are compatible (rolling-upgrade ordering).
3. Identify the serving datanode from the exception/trace and check its logs; if one node is always involved, restart or isolate it.
4. hdfs fsck the affected file to rule out genuine block corruption.

try {
  return readBlockFully(dfs, path, offset, len);
} catch (IOException e) {
  if (e.getMessage() != null
      && e.getMessage().contains("Expected empty end-of-read packet")) {
    // end-of-block handshake violation: reopen; a new replica usually works
    return readBlockFully(dfs, path, offset, len);
  }
  throw e;
}

When it happens

Trigger: Reading to the end of a block where the serving datanode emits a malformed trailer: typically version/protocol skew between client and datanode, a datanode bug, or a corrupted stream whose garbage is parsed as the trailer header.

Common situations: Mixed-version clusters during rolling upgrades; patched or vendor datanodes with non-conformant trailer behavior; environments that also produce packet-header sanity failures (bad NICs, offload bugs).

Related errors

• BlockReader: error in packet header {}
• Invalid payload length ${payloadLen}
• Invalid packet: data length in packet header exceeds data le
• Unsupported protocol found when creating the proxy connectio
• Version Mismatch (Expected: {}, Received: {} )

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/90cc1d1e4e70f490. Report an issue: GitHub.