{"record":{"id":"fb077cd53fe714e9","repo":"apache/hadoop","slug":"checksum-type-not-matched-for-file-filename","errorCode":null,"errorMessage":"Checksum ${type} not matched for file ${filename} at position ${errPos}: expected=%X but computed=%X, algorithm=${algorithmClass}","messagePattern":"Checksum (.+?) not matched for file (.+?) at position (.+?): expected=%X but computed=%X, algorithm=(.+?)","errorType":"exception","errorClass":"ChecksumException","httpStatus":null,"severity":"critical","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java","lineNumber":493,"sourceCode":"    if (remainder > 0) {\n      algorithm.reset();\n      algorithm.update(data, i, remainder);\n      final int computed = (int)algorithm.getValue();\n      final int expected = ((crcs[j] << 24) + ((crcs[j + 1] << 24) >>> 8))\n          + (((crcs[j + 2] << 24) >>> 16) + ((crcs[j + 3] << 24) >>> 24));\n\n      if (computed != expected) {\n        final long errPos = basePos + i - dataOffset;\n        throwChecksumException(type, algorithm, filename, errPos, expected,\n            computed);\n      }\n    }\n  }\n\n  private static void throwChecksumException(Type type, Checksum algorithm,\n      String filename, long errPos, int expected, int computed)\n          throws ChecksumException {\n    throw new ChecksumException(\"Checksum \" + type\n        + \" not matched for file \" + filename + \" at position \"+ errPos\n        + String.format(\": expected=%X but computed=%X\", expected, computed)\n        + \", algorithm=\" + algorithm.getClass().getSimpleName(), errPos);\n  }\n\n  /**\n   * Calculate checksums for the given data.\n   * \n   * The 'mark' of the ByteBuffer parameters may be modified by this function,\n   * but the position is maintained.\n   * \n   * @param data the DirectByteBuffer pointing to the data to checksum.\n   * @param checksums the DirectByteBuffer into which checksums will be\n   *                  stored. Enough space must be available in this\n   *                  buffer to put the checksums.\n   */\n  public void calculateChunkedSums(ByteBuffer data, ByteBuffer checksums) {\n    if (type.size == 0) return;","sourceCodeStart":475,"sourceCodeEnd":511,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java#L475-L511","documentation":"DataChecksum.verifyChecksums recomputes each chunk's checksum over the data and compares it with the stored value; on the first mismatch it throws ChecksumException naming the checksum type, the file, the absolute position of the failing chunk (basePos + i - dataOffset), expected vs computed values in hex, and the algorithm class (e.g. CRC32). This is Hadoop's primary data-corruption signal — the bytes read no longer match their stored checksums.","triggerScenarios":"Reading a LocalFileSystem (ChecksumFileSystem) file whose .crc sidecar disagrees with the data — file modified outside Hadoop or copied without its .crc; a corrupted HDFS block chunk; bit rot or a partial write leaving data and checksums out of sync.","commonSituations":"Files rewritten in place by external tools leaving a stale .crc; scp/rsync copying data files without the .crc sidecars; failing disks; interrupted transfers.","solutions":["Recreate the file from a good source with `hadoop fs -put` so the data and .crc regenerate together","If the data file is known-good but the .crc is stale, delete the .crc sidecar and re-copy/rewrite the file","For HDFS, run `hdfs fsck` to locate and repair corrupted blocks via re-replication","As a lossy last resort for irreplaceable local files, set io.skip.checksum.errors=true to skip bad chunks"],"exampleFix":"# before: data copied without its checksum sidecar -> checksum mismatch on read\nscp host:/data/part-0000 /dfs/data/\n# after: copy through Hadoop so file and .crc stay consistent\nhadoop fs -put part-0000 /data/part-0000","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try (FSDataInputStream in = fs.open(path)) {\n  in.readFully(buf);\n} catch (org.apache.hadoop.fs.ChecksumException e) {\n  long badPos = e.getPos();\n  // data and stored checksums disagree at badPos: re-fetch from a good source or another replica\n  restoreFromBackup(path);\n}","preventionTips":["Move files with hadoop fs -put/-cp so data and .crc stay consistent","Never edit data files in place on LocalFileSystem, and never copy data without its .crc sidecars","Verify critical datasets periodically with `hadoop fs -checksum`"],"tags":["checksum","data-corruption","local-filesystem","crc","checksum-exception"],"backgroundTag":"checksum-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}