{"record":{"id":"eae769b2dbdd5e3c","repo":"apache/hadoop","slug":"checksum-error-at-eae769","errorCode":null,"errorMessage":"Checksum error: {} at {}","messagePattern":"Checksum error: (.+?) at (.+?)","errorType":"exception","errorClass":"ChecksumException","httpStatus":null,"severity":"critical","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFs.java","lineNumber":264,"sourceCode":"        int sumLenRead = sums.read(checksum, 0, CHECKSUM_SIZE * checksumsToRead);\n        if (sumLenRead >= 0 && sumLenRead % CHECKSUM_SIZE != 0) {\n          throw new EOFException(\"Checksum file not a length multiple of checksum size \" +\n                                 \"in \" + file + \" at \" + pos + \" checksumpos: \" + checksumPos +\n                                 \" sumLenread: \" + sumLenRead );\n        }\n        if (sumLenRead <= 0) { // we're at the end of the file\n          eof = true;\n        } else {\n          // Adjust amount of data to read based on how many checksum chunks we read\n          len = Math.min(len, bytesPerSum * (sumLenRead / CHECKSUM_SIZE));\n        }\n      }\n      if (pos != datas.getPos()) {\n        datas.seek(pos);\n      }\n      int nread = readFully(datas, buf, offset, len);\n      if (eof && nread > 0) {\n        throw new ChecksumException(\"Checksum error: \"+file+\" at \"+pos, pos);\n      }\n      return nread;\n    }\n    \n    /* Return the file length */\n    private long getFileLength() throws IOException, UnresolvedLinkException {\n      if (fileLen==-1L) {\n        fileLen = fs.getFileStatus(file).getLen();\n      }\n      return fileLen;\n    }\n    \n    /**\n     * Skips over and discards <code>n</code> bytes of data from the\n     * input stream.\n     *\n     * The <code>skip</code> method skips over some smaller number of bytes\n     * when reaching end of file before <code>n</code> bytes have been skipped.","sourceCodeStart":246,"sourceCodeEnd":282,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFs.java#L246-L282","documentation":"ChecksumFs's reader found data bytes past the end of checksum coverage (eof && nread > 0): the .crc sidecar ended before the data file did, so bytes exist that have no checksum to verify against. It surfaces as ChecksumException with the file and byte position. The usual root cause is that the data file grew (raw append) or its checksum file shrank/aged, i.e., the two are out of sync.","triggerScenarios":"Reading a file through FileContext on file:// after the data file was modified outside the checksum layer: appended via RawLocalFileSystem while the .crc stayed stale, .crc deleted from a middle-generation write, or the pair restored/copy-split inconsistently. The exception is thrown from read() when pos reaches the uncovered region.","commonSituations":"Bypassing ChecksumFs's 'append/truncate not supported' by writing through the raw fs and forgetting to remove the .crc; mixing hadoop fs -put with plain cp/rsync into the same directory; bit-rot on the volume corrupting the .crc itself; changing io.bytes.per.checksum between write and read of the same file pair.","solutions":["Confirm the diagnosis: fc.setVerifyChecksum(false) and read the tail - if it succeeds the data survived and only checksum coverage is missing.","Delete the stale .crc sidecar (hidden dot-file, same dir: /p/.f.crc) so reads skip verification until the next create rebuilds it.","If actual corruption is suspected (reads still fail unverified), restore from a replica/backup or recopy the source with hadoop fs -put so a fresh .crc is generated.","Stop appending through the raw filesystem, or always delete the .crc after such writes, to prevent recurrence."],"exampleFix":"// before\nFSDataInputStream in = fc.open(path);\nin.read(buffer); // ChecksumException: Checksum error: /data/f at 65536\n\n// after\nfc.setVerifyChecksum(false);\nFSDataInputStream in = fc.open(path);\nin.readFully(buffer); // succeeds if only the .crc is stale\njava.nio.file.Files.deleteIfExists(java.nio.file.Paths.get(\"/data/.f.crc\"));","handlingStrategy":"fallback","validationCode":"fc.setVerifyChecksum(false); // read-side bypass when .crc coverage is known-stale\n// long dataLen = fc.open(path).getPos(); // stream length still works unverified","typeGuard":null,"tryCatchPattern":"try {\n  readVerified(fc, path);\n} catch (ChecksumException e) {\n  // triage: fc.setVerifyChecksum(false); delete stale .crc; restore from replica if data truly bad\n}","preventionTips":["Route all writes to checksummed local paths through the checksummed fs handle.","Delete .crc sidecars whenever you bypass the checksum layer for append/truncate.","Copy file+.crc pairs atomically; never split them across backup restores."],"tags":["hadoop","filesystem","checksum","checksum-exception","data-corruption","local-filesystem"],"backgroundTag":"checksum-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}