{"record":{"id":"90cc1d1e4e70f490","repo":"apache/hadoop","slug":"expected-empty-end-of-read-packet-header","errorCode":null,"errorMessage":"Expected empty end-of-read packet! Header: {}","messagePattern":"Expected empty end-of-read packet! Header: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java","lineNumber":273,"sourceCode":"        break;\n      }\n\n      int skip = (int)Math.min(curDataSlice.remaining(), needToSkip);\n      curDataSlice.position(curDataSlice.position() + skip);\n      skipped += skip;\n    }\n    return skipped;\n  }\n\n  private void readTrailingEmptyPacket() throws IOException {\n    LOG.trace(\"Reading empty packet at end of read\");\n\n    packetReceiver.receiveNextPacket(in);\n\n    PacketHeader trailer = packetReceiver.getHeader();\n    if (!trailer.isLastPacketInBlock() ||\n        trailer.getDataLen() != 0) {\n      throw new IOException(\"Expected empty end-of-read packet! Header: \" +\n          trailer);\n    }\n  }\n\n  protected BlockReaderRemote(String file, long blockId,\n                              DataChecksum checksum, boolean verifyChecksum,\n                              long startOffset, long firstChunkOffset,\n                              long bytesToRead, Peer peer,\n                              DatanodeID datanodeID, PeerCache peerCache,\n                              int networkDistance) {\n    // Path is used only for printing block and file information in debug\n    this.peer = peer;\n    this.datanodeID = datanodeID;\n    this.in = peer.getInputStreamChannel();\n    this.checksum = checksum;\n    this.verifyChecksum = verifyChecksum;\n    this.startOffset = Math.max( startOffset, 0 );\n    this.filename = file;","sourceCodeStart":255,"sourceCodeEnd":291,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java#L255-L291","documentation":"When a client reaches the end of a block, the data-transfer protocol requires the datanode to send one final empty packet flagged lastPacketInBlock. readTrailingEmptyPacket throws when that trailer packet carries data or is not marked as the last packet — the datanode violated the end-of-block handshake.","triggerScenarios":"Reading to the end of a block where the serving datanode emits a malformed trailer: typically version/protocol skew between client and datanode, a datanode bug, or a corrupted stream whose garbage is parsed as the trailer header.","commonSituations":"Mixed-version clusters during rolling upgrades; patched or vendor datanodes with non-conformant trailer behavior; environments that also produce packet-header sanity failures (bad NICs, offload bugs).","solutions":["Retry the read; transient cases resolve on a new connection or replica.","Confirm client and datanode Hadoop versions are compatible (rolling-upgrade ordering).","Identify the serving datanode from the exception/trace and check its logs; if one node is always involved, restart or isolate it.","hdfs fsck the affected file to rule out genuine block corruption."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  return readBlockFully(dfs, path, offset, len);\n} catch (IOException e) {\n  if (e.getMessage() != null\n      && e.getMessage().contains(\"Expected empty end-of-read packet\")) {\n    // end-of-block handshake violation: reopen; a new replica usually works\n    return readBlockFully(dfs, path, offset, len);\n  }\n  throw e;\n}","preventionTips":["Make large reads retryable with a reopen-from-offset wrapper.","Track client vs datanode Hadoop versions; protocol skew shows up exactly here.","If a single datanode keeps producing trailer errors, take it out of rotation and check its logs."],"tags":["hdfs","data-transfer","packet","end-of-block","protocol"],"backgroundTag":"data-transfer-protocol-error","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}