{"record":{"id":"879de3a4b5c1143a","repo":"apache/hadoop","slug":"offset-startoffset-and-length-length-don-t-ma","errorCode":null,"errorMessage":" Offset {startOffset} and length {length} don't match block {block} ( blockLen {end} )","messagePattern":" Offset (.+?) and length (.+?) don't match block (.+?) \\( blockLen (.+?) \\)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java","lineNumber":399,"sourceCode":"            Math.max((int)replicaVisibleLength, 10*1024*1024));\n        size = csum.getBytesPerChecksum();        \n      }\n      chunkSize = size;\n      checksum = csum;\n      checksumSize = checksum.getChecksumSize();\n      length = length < 0 ? replicaVisibleLength : length;\n\n      // end is either last byte on disk or the length for which we have a \n      // checksum\n      long end = chunkChecksum != null ? chunkChecksum.getDataLength()\n          : replica.getBytesOnDisk();\n      if (startOffset < 0 || startOffset > end\n          || (length + startOffset) > end) {\n        String msg = \" Offset \" + startOffset + \" and length \" + length\n        + \" don't match block \" + block + \" ( blockLen \" + end + \" )\";\n        LOG.warn(datanode.getDNRegistrationForBP(block.getBlockPoolId()) +\n            \":sendBlock() : \" + msg);\n        throw new IOException(msg);\n      }\n      \n      // Ensure read offset is position at the beginning of chunk\n      offset = startOffset - (startOffset % chunkSize);\n      if (length >= 0) {\n        // Ensure endOffset points to end of chunk.\n        long tmpLen = startOffset + length;\n        if (tmpLen % chunkSize != 0) {\n          tmpLen += (chunkSize - tmpLen % chunkSize);\n        }\n        if (tmpLen < end) {\n          // will use on-disk checksum here since the end is a stable chunk\n          end = tmpLen;\n        } else if (chunkChecksum != null) {\n          // last chunk is changing. flag that we need to use in-memory checksum \n          this.lastChunkChecksum = chunkChecksum;\n        }\n      }","sourceCodeStart":381,"sourceCodeEnd":417,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java#L381-L417","documentation":"BlockSender validates the requested byte range against 'end' — the data length covered by checksums (chunkChecksum.getDataLength()) when known, otherwise bytesOnDisk. If startOffset is negative, startOffset exceeds end, or startOffset+length exceeds end, the requested window is outside what the datanode can safely read, and the IOException (also WARN-logged with sendBlock context) aborts the transfer.","triggerScenarios":"A readBlock op whose offset/length were computed from a stale block length: client opened a file, the block grew (append) or its visible length shrank on this replica, then the read used old numbers; or a caller passed a negative/garbage offset; or reading beyond the checksummed length of an RBW replica during hflush/hsync races.","commonSituations":"Long-lived DFSInputStream with stale located-block info reading concurrently-written files; custom MapReduce/s3a-style readers computing ranges from cached lengths; offset math bugs that request length bytes past EOF.","solutions":["Client-side: clamp (offset, length) to the LocatedBlock length fetched from the NameNode before issuing the read, and re-fetch locations on this error before retrying.","For custom tooling over HSFTP/WebHDFS/data-transfer, validate offset >= 0 and offset+length <= block visible length.","If reading files under active appends, re-open or re-fetch block locations when the range check trips instead of retrying the same stale range.","Verify with fsck that the block's length is consistent across replicas if ranges keep failing."],"exampleFix":"// before: read with stale cached length\nlong len = cachedLocatedBlock.getBlockSize();\nin.readBlock(block, token, clientName, offset, len);\n\n// after: clamp requested range to freshly located length\nLocatedBlock lb = namenode.getBlockLocations(file, offset, 1).get(0);\nlong available = Math.max(0, lb.getBlockSize() - offset);\nlong readLen = Math.min(requestedLen, available);\nin.readBlock(lb.getBlock(), token, clientName, offset, readLen);","handlingStrategy":"validation","validationCode":"// Clamp the read range to the located block length before transferring\nLocatedBlock lb = namenode.getBlockLocations(file, offset, 1).get(0);\nlong blockVisible = lb.getBlockSize();\nif (offset < 0 || offset > blockVisible) throw new IllegalArgumentException(\"bad offset\");\nlong length = Math.min(requestedLength, blockVisible - offset);\n// then issue readBlock(offset, length)","typeGuard":null,"tryCatchPattern":"try {\n  in.readBlock(block, token, clientName, offset, length);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"don't match block\")) {\n    lb = namenode.getBlockLocations(file, offset, 1).get(0); // length changed\n    length = Math.min(length, lb.getBlockSize() - offset);\n    in.readBlock(lb.getBlock(), token, clientName, offset, length);\n  } else {\n    throw e;\n  }\n}","preventionTips":["Always bound offset+length by the LocatedBlock length from the NameNode, not a cached file length.","For files under active append, re-fetch block locations between reads.","Unit-test custom readers with offset==length and offset==blockSize edge cases before shipping."],"tags":["hdfs","datanode","read-path","offset","validation"],"backgroundTag":"invalid-read-offset","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}