apache/hadoop · error · IOException

Expected to read {checksumSize} bytes from offset {offsetInC

Error message

Expected to read {checksumSize} bytes from offset {offsetInChecksum} but read {readBytes} bytes.

What it means

Thrown by FsVolumeImpl.loadLastPartialChunkChecksum() (FsVolumeImpl.java:1256) when the seek into the meta file for the last partial chunk's checksum succeeded but read() returned fewer bytes than checksumSize (not -1). A partial read at a computed offset means the meta file is longer than nothing but still inconsistent with the block file length - typically a truncated or damaged meta file, or a checksum-size mismatch between the header and the data actually stored.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java:1256

    if (onDiskLen % bytesPerChecksum == 0) {
      // the last chunk is a complete one. No need to preserve its checksum
      // because it will not be modified.
      return null;
    }

    long offsetInChecksum = BlockMetadataHeader.getHeaderSize() +
        (onDiskLen / bytesPerChecksum) * checksumSize;
    byte[] lastChecksum = new byte[checksumSize];
    try (RandomAccessFile raf = fileIoProvider.getRandomAccessFile(
        this, metaFile, "r")) {
      raf.seek(offsetInChecksum);
      int readBytes = raf.read(lastChecksum, 0, checksumSize);
      if (readBytes == -1) {
        throw new IOException("Expected to read " + checksumSize +
            " bytes from offset " + offsetInChecksum +
            " but reached end of file.");
      } else if (readBytes != checksumSize) {
        throw new IOException("Expected to read " + checksumSize +
            " bytes from offset " + offsetInChecksum + " but read " +
            readBytes + " bytes.");
      }
    }
    return lastChecksum;
  }

  public ReplicaInPipeline append(String bpid, ReplicaInfo replicaInfo,
      long newGS, long estimateBlockLen) throws IOException {

    long bytesReserved = estimateBlockLen - replicaInfo.getNumBytes();
    if (getAvailable() < bytesReserved) {
      throw new DiskOutOfSpaceException("Insufficient space for appending to "
          + replicaInfo);
    }

    assert replicaInfo.getVolume() == this:
      "The volume of the replica should be the same as this volume";

View on GitHub (pinned to 2add963021)

Solutions

Validate the meta/block size relationship (expected meta = headerSize + ceil(blockLen/bytesPerChecksum)*checksumSize) and treat mismatches as replica corruption
Remove the bad replica so it re-replicates from a healthy peer, then retry the truncate
Run hdfs fsck /files to confirm which replicas are damaged before deleting anything
If many replicas show this on one volume, suspect the disk (smartctl/dmesg) and retire the volume

Example fix

// before: partial read at offset surfaces as opaque IOException
int readBytes = raf.read(lastChecksum, 0, checksumSize);

// after: detect inconsistency up front and fail with an actionable message
long expected = BlockMetadataHeader.getHeaderSize()
    + ((onDiskLen + bytesPerChecksum - 1) / bytesPerChecksum) * checksumSize;
if (metaFile.length() != expected) {
  throw new IOException("Corrupt replica " + blockFile
      + ": meta length " + metaFile.length() + " != expected " + expected);
}

Defensive patterns

Strategy: try-catch

Validate before calling

long expectedMeta = BlockMetadataHeader.getHeaderSize()
    + ((blockFile.length() + bytesPerChecksum - 1) / bytesPerChecksum)
      * checksumSize;
if (metaFile.length() != expectedMeta) {
  // partial reads at the computed offset are guaranteed once lengths diverge
  LOG.warn("Skipping replica {}: meta {} != expected {}",
      blockFile, metaFile.length(), expectedMeta);
}

Try / catch

try {
  volume.loadLastPartialChunkChecksum(blockFile, metaFile);
} catch (IOException e) {
  // size-inconsistent replica: invalidate so a healthy copy re-replicates instead of hand-repairing
  dataset.invalidate(bpid, new Block[] {block});
  throw e;
}

Prevention

Keep block/meta pairs immutable outside HDFS; no manual padding or truncation
Use consistent snapshots/backups so block and meta files match
fsck after any storage incident before running truncate workloads

When it happens

Trigger: Truncate flow on a replica whose meta file lost its final checksum bytes (crash mid-checksum-write, bit rot); meta header declares a checksum type whose size differs from the trailing data layout; block file length edited/restored without matching meta regeneration.

Common situations: Post-crash replica recovery where the meta file write did not complete; volumes restored from inconsistent snapshots; hardware corruption affecting only the tail of meta files.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ea101a2f0f04bdfb. Report an issue: GitHub.