{"record":{"id":"ea101a2f0f04bdfb","repo":"apache/hadoop","slug":"expected-to-read-checksumsize-bytes-from-offset-ea101a","errorCode":null,"errorMessage":"Expected to read {checksumSize} bytes from offset {offsetInChecksum} but read {readBytes} bytes.","messagePattern":"Expected to read (.+?) bytes from offset (.+?) but read (.+?) bytes\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java","lineNumber":1256,"sourceCode":"    if (onDiskLen % bytesPerChecksum == 0) {\n      // the last chunk is a complete one. No need to preserve its checksum\n      // because it will not be modified.\n      return null;\n    }\n\n    long offsetInChecksum = BlockMetadataHeader.getHeaderSize() +\n        (onDiskLen / bytesPerChecksum) * checksumSize;\n    byte[] lastChecksum = new byte[checksumSize];\n    try (RandomAccessFile raf = fileIoProvider.getRandomAccessFile(\n        this, metaFile, \"r\")) {\n      raf.seek(offsetInChecksum);\n      int readBytes = raf.read(lastChecksum, 0, checksumSize);\n      if (readBytes == -1) {\n        throw new IOException(\"Expected to read \" + checksumSize +\n            \" bytes from offset \" + offsetInChecksum +\n            \" but reached end of file.\");\n      } else if (readBytes != checksumSize) {\n        throw new IOException(\"Expected to read \" + checksumSize +\n            \" bytes from offset \" + offsetInChecksum + \" but read \" +\n            readBytes + \" bytes.\");\n      }\n    }\n    return lastChecksum;\n  }\n\n  public ReplicaInPipeline append(String bpid, ReplicaInfo replicaInfo,\n      long newGS, long estimateBlockLen) throws IOException {\n\n    long bytesReserved = estimateBlockLen - replicaInfo.getNumBytes();\n    if (getAvailable() < bytesReserved) {\n      throw new DiskOutOfSpaceException(\"Insufficient space for appending to \"\n          + replicaInfo);\n    }\n\n    assert replicaInfo.getVolume() == this:\n      \"The volume of the replica should be the same as this volume\";","sourceCodeStart":1238,"sourceCodeEnd":1274,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java#L1238-L1274","documentation":"Thrown by FsVolumeImpl.loadLastPartialChunkChecksum() (FsVolumeImpl.java:1256) when the seek into the meta file for the last partial chunk's checksum succeeded but read() returned fewer bytes than checksumSize (not -1). A partial read at a computed offset means the meta file is longer than nothing but still inconsistent with the block file length - typically a truncated or damaged meta file, or a checksum-size mismatch between the header and the data actually stored.","triggerScenarios":"Truncate flow on a replica whose meta file lost its final checksum bytes (crash mid-checksum-write, bit rot); meta header declares a checksum type whose size differs from the trailing data layout; block file length edited/restored without matching meta regeneration.","commonSituations":"Post-crash replica recovery where the meta file write did not complete; volumes restored from inconsistent snapshots; hardware corruption affecting only the tail of meta files.","solutions":["Validate the meta/block size relationship (expected meta = headerSize + ceil(blockLen/bytesPerChecksum)*checksumSize) and treat mismatches as replica corruption","Remove the bad replica so it re-replicates from a healthy peer, then retry the truncate","Run hdfs fsck /files to confirm which replicas are damaged before deleting anything","If many replicas show this on one volume, suspect the disk (smartctl/dmesg) and retire the volume"],"exampleFix":"// before: partial read at offset surfaces as opaque IOException\nint readBytes = raf.read(lastChecksum, 0, checksumSize);\n\n// after: detect inconsistency up front and fail with an actionable message\nlong expected = BlockMetadataHeader.getHeaderSize()\n    + ((onDiskLen + bytesPerChecksum - 1) / bytesPerChecksum) * checksumSize;\nif (metaFile.length() != expected) {\n  throw new IOException(\"Corrupt replica \" + blockFile\n      + \": meta length \" + metaFile.length() + \" != expected \" + expected);\n}","handlingStrategy":"try-catch","validationCode":"long expectedMeta = BlockMetadataHeader.getHeaderSize()\n    + ((blockFile.length() + bytesPerChecksum - 1) / bytesPerChecksum)\n      * checksumSize;\nif (metaFile.length() != expectedMeta) {\n  // partial reads at the computed offset are guaranteed once lengths diverge\n  LOG.warn(\"Skipping replica {}: meta {} != expected {}\",\n      blockFile, metaFile.length(), expectedMeta);\n}","typeGuard":null,"tryCatchPattern":"try {\n  volume.loadLastPartialChunkChecksum(blockFile, metaFile);\n} catch (IOException e) {\n  // size-inconsistent replica: invalidate so a healthy copy re-replicates instead of hand-repairing\n  dataset.invalidate(bpid, new Block[] {block});\n  throw e;\n}","preventionTips":["Keep block/meta pairs immutable outside HDFS; no manual padding or truncation","Use consistent snapshots/backups so block and meta files match","fsck after any storage incident before running truncate workloads"],"tags":["hdfs","datanode","checksum","truncation","corruption"],"backgroundTag":"corrupt-block-metadata-file","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}