{"record":{"id":"f8d23917736d876f","repo":"apache/hadoop","slug":"expected-to-read-checksumsize-bytes-from-offset","errorCode":null,"errorMessage":"Expected to read {checksumSize} bytes from offset {offsetInChecksum} but reached end of file.","messagePattern":"Expected to read (.+?) bytes from offset (.+?) but reached end of file\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java","lineNumber":1252,"sourceCode":"    final int checksumSize = dcs.getChecksumSize();\n    final long onDiskLen = blockFile.length();\n    final int bytesPerChecksum = dcs.getBytesPerChecksum();\n\n    if (onDiskLen % bytesPerChecksum == 0) {\n      // the last chunk is a complete one. No need to preserve its checksum\n      // because it will not be modified.\n      return null;\n    }\n\n    long offsetInChecksum = BlockMetadataHeader.getHeaderSize() +\n        (onDiskLen / bytesPerChecksum) * checksumSize;\n    byte[] lastChecksum = new byte[checksumSize];\n    try (RandomAccessFile raf = fileIoProvider.getRandomAccessFile(\n        this, metaFile, \"r\")) {\n      raf.seek(offsetInChecksum);\n      int readBytes = raf.read(lastChecksum, 0, checksumSize);\n      if (readBytes == -1) {\n        throw new IOException(\"Expected to read \" + checksumSize +\n            \" bytes from offset \" + offsetInChecksum +\n            \" but reached end of file.\");\n      } else if (readBytes != checksumSize) {\n        throw new IOException(\"Expected to read \" + checksumSize +\n            \" bytes from offset \" + offsetInChecksum + \" but read \" +\n            readBytes + \" bytes.\");\n      }\n    }\n    return lastChecksum;\n  }\n\n  public ReplicaInPipeline append(String bpid, ReplicaInfo replicaInfo,\n      long newGS, long estimateBlockLen) throws IOException {\n\n    long bytesReserved = estimateBlockLen - replicaInfo.getNumBytes();\n    if (getAvailable() < bytesReserved) {\n      throw new DiskOutOfSpaceException(\"Insufficient space for appending to \"\n          + replicaInfo);","sourceCodeStart":1234,"sourceCodeEnd":1270,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java#L1234-L1270","documentation":"Thrown by FsVolumeImpl.loadLastPartialChunkChecksum() (FsVolumeImpl.java:1252). It reads the checksum of the last partial chunk from a replica's meta file by seeking to headerSize + (onDiskLen/bytesPerChecksum)*checksumSize; hitting EOF means the meta file is shorter than the block file implies - the block/meta pair is truncated or out of sync. This method backs the block truncation path, so the error surfaces as 'truncate' failing on the DataNode.","triggerScenarios":"FSTruncate on a replica whose meta file was truncated (crash during write, disk corruption) while the block file kept its length; block file extended (e.g. manually restored/padded) without regenerating the meta file; meta file from an older layout with a different checksum type/size than the header now reports.","commonSituations":"Recovering a volume from backup/snapshot where block and meta files are from different points in time; silent disk corruption shrinking the meta file; mixing replicas restored with different bytesPerChecksum settings (e.g. after changing checksum type defaults).","solutions":["Compare sizes: expected meta length = BlockMetadataHeader.getHeaderSize() + ceil(blockFile.length()/bytesPerChecksum)*checksumSize; if the actual meta file is short, the replica is corrupt","Delete the corrupt replica so the NameNode re-replicates it (hdfs fsck -delete after confirming), then retry the truncate","If the block file is the padded/restored one, remove the stale pair and let re-replication rebuild it rather than repairing by hand","Check the volume for hardware errors (dmesg, smartctl) before trusting other replicas on it"],"exampleFix":"// before: trusting the pair and calling truncate blindly\nbyte[] last = volume.loadLastPartialChunkChecksum(blockFile, metaFile);\n\n// after: validate meta length matches block length first\nDataChecksum dcs = BlockMetadataHeader.readHeader(\n    new FileInputStream(metaFile)).getChecksum();\nlong expected = BlockMetadataHeader.getHeaderSize()\n    + ((blockFile.length() + dcs.getBytesPerChecksum() - 1)\n       / dcs.getBytesPerChecksum()) * dcs.getChecksumSize();\nif (metaFile.length() < expected) {\n  // replica is corrupt: invalidate and let NameNode re-replicate\n  throw new IOException(\"Corrupt replica \" + blockFile\n      + \" meta short: \" + metaFile.length() + \" < \" + expected);\n}","handlingStrategy":"try-catch","validationCode":"// expected meta length for this block file; mismatch means do not even attempt the read\nDataChecksum dcs = BlockMetadataHeader.readHeader(\n    new FileInputStream(metaFile)).getChecksum();\nlong expectedMeta = BlockMetadataHeader.getHeaderSize()\n    + ((blockFile.length() + dcs.getBytesPerChecksum() - 1)\n       / dcs.getBytesPerChecksum()) * dcs.getChecksumSize();\nif (metaFile.length() < expectedMeta) {\n  throw new IOException(\"Corrupt replica \" + blockFile\n      + \": meta shorter than block implies (\"\n      + metaFile.length() + \" < \" + expectedMeta + \")\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  byte[] lastCsum = volume.loadLastPartialChunkChecksum(blockFile, metaFile);\n} catch (IOException e) {\n  // meta/block mismatch: report the replica corrupt so the NN re-replicates, then fail the truncate\n  LOG.error(\"Replica {}/{} inconsistent: {}\", blockFile, metaFile, e);\n  dataset.invalidate(bpid, new Block[] {block});\n  throw e;\n}","preventionTips":["Never restore block and meta files from different points in time (consistent snapshots only)","Monitor disk health so truncation is caught before truncate/cache operations hit it","Run periodic hdfs fsck to catch replica corruption early"],"tags":["hdfs","datanode","checksum","truncation","corruption"],"backgroundTag":"corrupt-block-metadata-file","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}