{"record":{"id":"c240921065e716e1","repo":"apache/hadoop","slug":"got-unexpected-checksum-file-eof-at-block-file","errorCode":null,"errorMessage":"Got unexpected checksum file EOF at {}, block file position {} for block {} of file {}","messagePattern":"Got unexpected checksum file EOF at (.+?), block file position (.+?) for block (.+?) of file (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderLocal.java","lineNumber":378,"sourceCode":"    if (canSkipChecksum) {\n      freeChecksumBufIfExists();\n      return total;\n    }\n    if (total > 0) {\n      try {\n        buf.limit(buf.position());\n        buf.position(startBufPos);\n        createChecksumBufIfNeeded();\n        int checksumsNeeded = (total + bytesPerChecksum - 1) /\n            bytesPerChecksum;\n        checksumBuf.clear();\n        checksumBuf.limit(checksumsNeeded * checksumSize);\n        long checksumPos = BlockMetadataHeader.getHeaderSize()\n            + ((startDataPos / bytesPerChecksum) * checksumSize);\n        while (checksumBuf.hasRemaining()) {\n          int nRead = checksumIn.read(checksumBuf, checksumPos);\n          if (nRead < 0) {\n            throw new IOException(\"Got unexpected checksum file EOF at \" +\n                checksumPos + \", block file position \" + startDataPos +\n                \" for block \" + block + \" of file \" + filename);\n          }\n          checksumPos += nRead;\n        }\n        checksumBuf.flip();\n\n        checksum.verifyChunkedSums(buf, checksumBuf, filename, startDataPos);\n      } finally {\n        buf.position(buf.limit());\n      }\n    }\n    return total;\n  }\n\n  private boolean createNoChecksumContext() {\n    return !verifyChecksum ||\n        // Checksums are not stored for replicas on transient storage.  We do","sourceCodeStart":360,"sourceCodeEnd":396,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderLocal.java#L360-L396","documentation":"BlockReaderLocal performs short-circuit reads directly from the datanode's block and .meta files, verifying checksums locally. Here it computed exactly how many checksum bytes the .meta file must contain for the data range being read, and read() on the metadata stream returned EOF before that many bytes arrived — the checksum file is shorter than the data file implies, i.e., the local replica is corrupt or truncated on disk.","triggerScenarios":"A short-circuit read of a block whose .meta (checksum) file is truncated relative to the block data — after disk-full crashes, partial writes, or on-disk corruption. The client is co-located with that datanode, which is why only some nodes see it.","commonSituations":"Failing disks on datanodes; interrupted block writes leaving inconsistent block/meta pairs; corruption that surfaces only after short-circuit reads are enabled because remote reads pick other replicas.","solutions":["Run 'hdfs fsck /path -files -blocks -locations' to confirm which replica is corrupt.","Force re-replication of the bad replica: stop the affected datanode briefly, or use 'hdfs fsck -delete'/'-move' once corruption is confirmed.","As an immediate workaround set dfs.client.read.shortcircuit=false so clients read a healthy remote replica.","Inspect the datanode host: dmesg/SMART errors, full disks, and datanode logs for block write failures."],"exampleFix":"# diagnose\nhdfs fsck /path/to/file -files -blocks -locations\n# heal once corruption is confirmed (drops corrupt replicas, triggers re-replication)\nhdfs fsck /path/to/file -delete","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  return readLocal(dfs, path); // short-circuit path\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"checksum file EOF\")) {\n    // local replica is corrupt: read a remote replica and alert ops\n    conf.setBoolean(\"dfs.client.read.shortcircuit\", false);\n    alertOps(\"Possible corrupt replica for \" + path);\n    return readNetwork(dfs, path);\n  }\n  throw e;\n}","preventionTips":["Schedule periodic 'hdfs fsck' so corrupt replicas are found before readers hit them.","Monitor datanode disk health (SMART, dmesg) and full-disk conditions.","Treat short-circuit checksum EOF errors as replica health signals, not client bugs — they often expose corruption remote reads mask."],"tags":["hdfs","block-corruption","short-circuit-read","checksums","metadata-file"],"backgroundTag":"block-checksum-corruption","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}