apache/hadoop · error · IOException

Expected to read {checksumSize} bytes from offset {offsetInC

Error message

Expected to read {checksumSize} bytes from offset {offsetInChecksum} but reached end of file.

What it means

Thrown by FsVolumeImpl.loadLastPartialChunkChecksum() (FsVolumeImpl.java:1252). It reads the checksum of the last partial chunk from a replica's meta file by seeking to headerSize + (onDiskLen/bytesPerChecksum)*checksumSize; hitting EOF means the meta file is shorter than the block file implies - the block/meta pair is truncated or out of sync. This method backs the block truncation path, so the error surfaces as 'truncate' failing on the DataNode.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java:1252

    final int checksumSize = dcs.getChecksumSize();
    final long onDiskLen = blockFile.length();
    final int bytesPerChecksum = dcs.getBytesPerChecksum();

    if (onDiskLen % bytesPerChecksum == 0) {
      // the last chunk is a complete one. No need to preserve its checksum
      // because it will not be modified.
      return null;
    }

    long offsetInChecksum = BlockMetadataHeader.getHeaderSize() +
        (onDiskLen / bytesPerChecksum) * checksumSize;
    byte[] lastChecksum = new byte[checksumSize];
    try (RandomAccessFile raf = fileIoProvider.getRandomAccessFile(
        this, metaFile, "r")) {
      raf.seek(offsetInChecksum);
      int readBytes = raf.read(lastChecksum, 0, checksumSize);
      if (readBytes == -1) {
        throw new IOException("Expected to read " + checksumSize +
            " bytes from offset " + offsetInChecksum +
            " but reached end of file.");
      } else if (readBytes != checksumSize) {
        throw new IOException("Expected to read " + checksumSize +
            " bytes from offset " + offsetInChecksum + " but read " +
            readBytes + " bytes.");
      }
    }
    return lastChecksum;
  }

  public ReplicaInPipeline append(String bpid, ReplicaInfo replicaInfo,
      long newGS, long estimateBlockLen) throws IOException {

    long bytesReserved = estimateBlockLen - replicaInfo.getNumBytes();
    if (getAvailable() < bytesReserved) {
      throw new DiskOutOfSpaceException("Insufficient space for appending to "
          + replicaInfo);

View on GitHub (pinned to 2add963021)

Solutions

1. Compare sizes: expected meta length = BlockMetadataHeader.getHeaderSize() + ceil(blockFile.length()/bytesPerChecksum)*checksumSize; if the actual meta file is short, the replica is corrupt
2. Delete the corrupt replica so the NameNode re-replicates it (hdfs fsck -delete after confirming), then retry the truncate
3. If the block file is the padded/restored one, remove the stale pair and let re-replication rebuild it rather than repairing by hand
4. Check the volume for hardware errors (dmesg, smartctl) before trusting other replicas on it

Example fix

// before: trusting the pair and calling truncate blindly
byte[] last = volume.loadLastPartialChunkChecksum(blockFile, metaFile);

// after: validate meta length matches block length first
DataChecksum dcs = BlockMetadataHeader.readHeader(
    new FileInputStream(metaFile)).getChecksum();
long expected = BlockMetadataHeader.getHeaderSize()
    + ((blockFile.length() + dcs.getBytesPerChecksum() - 1)
       / dcs.getBytesPerChecksum()) * dcs.getChecksumSize();
if (metaFile.length() < expected) {
  // replica is corrupt: invalidate and let NameNode re-replicate
  throw new IOException("Corrupt replica " + blockFile
      + " meta short: " + metaFile.length() + " < " + expected);
}

// expected meta length for this block file; mismatch means do not even attempt the read
DataChecksum dcs = BlockMetadataHeader.readHeader(
    new FileInputStream(metaFile)).getChecksum();
long expectedMeta = BlockMetadataHeader.getHeaderSize()
    + ((blockFile.length() + dcs.getBytesPerChecksum() - 1)
       / dcs.getBytesPerChecksum()) * dcs.getChecksumSize();
if (metaFile.length() < expectedMeta) {
  throw new IOException("Corrupt replica " + blockFile
      + ": meta shorter than block implies ("
      + metaFile.length() + " < " + expectedMeta + ")");
}

try {
  byte[] lastCsum = volume.loadLastPartialChunkChecksum(blockFile, metaFile);
} catch (IOException e) {
  // meta/block mismatch: report the replica corrupt so the NN re-replicates, then fail the truncate
  LOG.error("Replica {}/{} inconsistent: {}", blockFile, metaFile, e);
  dataset.invalidate(bpid, new Block[] {block});
  throw e;
}

When it happens

Trigger: FSTruncate on a replica whose meta file was truncated (crash during write, disk corruption) while the block file kept its length; block file extended (e.g. manually restored/padded) without regenerating the meta file; meta file from an older layout with a different checksum type/size than the header now reports.

Common situations: Recovering a volume from backup/snapshot where block and meta files are from different points in time; silent disk corruption shrinking the meta file; mixing replicas restored with different bytesPerChecksum settings (e.g. after changing checksum type defaults).

Related errors

• Expected to read {checksumSize} bytes from offset {offsetInC
• checksum verification failed: premature EOF
• Checksum verification failed for the block ${blockFileName}:
• fetchBlockByteRange(). Got a checksum exception for {} at {}
• The block meta file header is corrupt

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f8d23917736d876f. Report an issue: GitHub.