apache/hadoop · error · IOException

Offset {startOffset} and length {length} don't match block

Error message

 Offset {startOffset} and length {length} don't match block {block} ( blockLen {end} )

What it means

BlockSender validates the requested byte range against 'end' — the data length covered by checksums (chunkChecksum.getDataLength()) when known, otherwise bytesOnDisk. If startOffset is negative, startOffset exceeds end, or startOffset+length exceeds end, the requested window is outside what the datanode can safely read, and the IOException (also WARN-logged with sendBlock context) aborts the transfer.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java:399

            Math.max((int)replicaVisibleLength, 10*1024*1024));
        size = csum.getBytesPerChecksum();        
      }
      chunkSize = size;
      checksum = csum;
      checksumSize = checksum.getChecksumSize();
      length = length < 0 ? replicaVisibleLength : length;

      // end is either last byte on disk or the length for which we have a 
      // checksum
      long end = chunkChecksum != null ? chunkChecksum.getDataLength()
          : replica.getBytesOnDisk();
      if (startOffset < 0 || startOffset > end
          || (length + startOffset) > end) {
        String msg = " Offset " + startOffset + " and length " + length
        + " don't match block " + block + " ( blockLen " + end + " )";
        LOG.warn(datanode.getDNRegistrationForBP(block.getBlockPoolId()) +
            ":sendBlock() : " + msg);
        throw new IOException(msg);
      }
      
      // Ensure read offset is position at the beginning of chunk
      offset = startOffset - (startOffset % chunkSize);
      if (length >= 0) {
        // Ensure endOffset points to end of chunk.
        long tmpLen = startOffset + length;
        if (tmpLen % chunkSize != 0) {
          tmpLen += (chunkSize - tmpLen % chunkSize);
        }
        if (tmpLen < end) {
          // will use on-disk checksum here since the end is a stable chunk
          end = tmpLen;
        } else if (chunkChecksum != null) {
          // last chunk is changing. flag that we need to use in-memory checksum 
          this.lastChunkChecksum = chunkChecksum;
        }
      }

View on GitHub (pinned to 2add963021)

Solutions

Client-side: clamp (offset, length) to the LocatedBlock length fetched from the NameNode before issuing the read, and re-fetch locations on this error before retrying.
For custom tooling over HSFTP/WebHDFS/data-transfer, validate offset >= 0 and offset+length <= block visible length.
If reading files under active appends, re-open or re-fetch block locations when the range check trips instead of retrying the same stale range.
Verify with fsck that the block's length is consistent across replicas if ranges keep failing.

Example fix

// before: read with stale cached length
long len = cachedLocatedBlock.getBlockSize();
in.readBlock(block, token, clientName, offset, len);

// after: clamp requested range to freshly located length
LocatedBlock lb = namenode.getBlockLocations(file, offset, 1).get(0);
long available = Math.max(0, lb.getBlockSize() - offset);
long readLen = Math.min(requestedLen, available);
in.readBlock(lb.getBlock(), token, clientName, offset, readLen);

Defensive patterns

Strategy: validation

Validate before calling

// Clamp the read range to the located block length before transferring
LocatedBlock lb = namenode.getBlockLocations(file, offset, 1).get(0);
long blockVisible = lb.getBlockSize();
if (offset < 0 || offset > blockVisible) throw new IllegalArgumentException("bad offset");
long length = Math.min(requestedLength, blockVisible - offset);
// then issue readBlock(offset, length)

Try / catch

try {
  in.readBlock(block, token, clientName, offset, length);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("don't match block")) {
    lb = namenode.getBlockLocations(file, offset, 1).get(0); // length changed
    length = Math.min(length, lb.getBlockSize() - offset);
    in.readBlock(lb.getBlock(), token, clientName, offset, length);
  } else {
    throw e;
  }
}

Prevention

Always bound offset+length by the LocatedBlock length from the NameNode, not a cached file length.
For files under active append, re-fetch block locations between reads.
Unit-test custom readers with offset==length and offset==blockSize edge cases before shipping.

When it happens

Trigger: A readBlock op whose offset/length were computed from a stale block length: client opened a file, the block grew (append) or its visible length shrank on this replica, then the read used old numbers; or a caller passed a negative/garbage offset; or reading beyond the checksummed length of an RBW replica during hflush/hsync races.

Common situations: Long-lived DFSInputStream with stale located-block info reading concurrently-written files; custom MapReduce/s3a-style readers computing ranges from cached lengths; offset math bugs that request length bytes past EOF.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/879de3a4b5c1143a. Report an issue: GitHub.