{"record":{"id":"70b1c533d621052f","repo":"apache/hadoop","slug":"blockreader-error-in-packet-header","errorCode":null,"errorMessage":"BlockReader: error in packet header {}","messagePattern":"BlockReader: error in packet header (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java","lineNumber":197,"sourceCode":"    buf.put(writeSlice);\n    curDataSlice.position(writeSlice.position());\n\n    return nRead;\n  }\n\n  private void readNextPacket() throws IOException {\n    //Read packet headers.\n    packetReceiver.receiveNextPacket(in);\n\n    PacketHeader curHeader = packetReceiver.getHeader();\n    curDataSlice = packetReceiver.getDataSlice();\n    assert curDataSlice.capacity() == curHeader.getDataLen();\n\n    LOG.trace(\"DFSClient readNextPacket got header {}\", curHeader);\n\n    // Sanity check the lengths\n    if (!curHeader.sanityCheck(lastSeqNo)) {\n      throw new IOException(\"BlockReader: error in packet header \" +\n          curHeader);\n    }\n\n    if (curHeader.getDataLen() > 0) {\n      int chunks = 1 + (curHeader.getDataLen() - 1) / bytesPerChecksum;\n      int checksumsLen = chunks * checksumSize;\n\n      assert packetReceiver.getChecksumSlice().capacity() == checksumsLen :\n          \"checksum slice capacity=\" +\n              packetReceiver.getChecksumSlice().capacity() +\n              \" checksumsLen=\" + checksumsLen;\n\n      lastSeqNo = curHeader.getSeqno();\n      if (verifyChecksum && curDataSlice.remaining() > 0) {\n        // N.B.: the checksum error offset reported here is actually\n        // relative to the start of the block, not the start of the file.\n        // This is slightly misleading, but preserves the behavior from\n        // the older BlockReader.","sourceCodeStart":179,"sourceCodeEnd":215,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java#L179-L215","documentation":"BlockReaderRemote receives a block as a sequence of packets; after parsing each header it calls PacketHeader.sanityCheck(lastSeqNo), which validates length, offset and sequence-number invariants against the previously seen packet. A header failing these checks produces this IOException — the packet stream itself is malformed.","triggerScenarios":"Any HDFS block read (open/read, MR/Spark splits, DistCp) where bytes on the data-transfer connection do not parse as a sane packet header: a corrupted TCP stream, a datanode with memory/GC problems emitting garbage, or protocol skew between client and datanode during a rolling upgrade.","commonSituations":"Faulty NICs or TCP-offload engines corrupting payloads; mixed client/datanode versions mid rolling-upgrade; long-lived connections traversing flaky interconnects, VPNs or LBs; datanodes under OOM/GC thrash.","solutions":["Retry the read — DFSInputStream re-fetches block locations and usually picks a different replica.","Run 'hdfs fsck /file -files -blocks -locations' to rule out real block corruption.","If reproducible on one host/path, inspect the network (disable NIC offload, swap cables) and the datanode's GC/OOM behavior and logs.","Verify client and datanode Hadoop versions are compatible during rolling upgrades."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"for (int attempt = 1; attempt <= 3; attempt++) {\n  try (FSDataInputStream in = dfs.open(path)) {\n    in.seek(lastGoodOffset);\n    return IOUtils.readFullyAsBytes(in, remaining);\n  } catch (IOException e) {\n    if (attempt == 3 || !isPacketHeaderError(e)) throw e;\n    lastGoodOffset = recomputeOffset(); // re-fetch locations on retry\n  }\n}","preventionTips":["Instrument read paths to record which datanode served a failing block — patterns point at one host or NIC.","Keep client and datanode versions aligned during rolling upgrades.","After repeated packet-header failures on one path, disable NIC TCP-offload on suspect hosts and retest."],"tags":["hdfs","data-transfer","packet","corruption","network"],"backgroundTag":"data-transfer-protocol-error","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}