apache/hadoop · critical · ChecksumException

Checksum ${type} not matched for file ${filename} at positio

Error message

Checksum ${type} not matched for file ${filename} at position ${errPos}: expected=%X but computed=%X, algorithm=${algorithmClass}

What it means

DataChecksum.verifyChecksums recomputes each chunk's checksum over the data and compares it with the stored value; on the first mismatch it throws ChecksumException naming the checksum type, the file, the absolute position of the failing chunk (basePos + i - dataOffset), expected vs computed values in hex, and the algorithm class (e.g. CRC32). This is Hadoop's primary data-corruption signal — the bytes read no longer match their stored checksums.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java:493

    if (remainder > 0) {
      algorithm.reset();
      algorithm.update(data, i, remainder);
      final int computed = (int)algorithm.getValue();
      final int expected = ((crcs[j] << 24) + ((crcs[j + 1] << 24) >>> 8))
          + (((crcs[j + 2] << 24) >>> 16) + ((crcs[j + 3] << 24) >>> 24));

      if (computed != expected) {
        final long errPos = basePos + i - dataOffset;
        throwChecksumException(type, algorithm, filename, errPos, expected,
            computed);
      }
    }
  }

  private static void throwChecksumException(Type type, Checksum algorithm,
      String filename, long errPos, int expected, int computed)
          throws ChecksumException {
    throw new ChecksumException("Checksum " + type
        + " not matched for file " + filename + " at position "+ errPos
        + String.format(": expected=%X but computed=%X", expected, computed)
        + ", algorithm=" + algorithm.getClass().getSimpleName(), errPos);
  }

  /**
   * Calculate checksums for the given data.
   * 
   * The 'mark' of the ByteBuffer parameters may be modified by this function,
   * but the position is maintained.
   * 
   * @param data the DirectByteBuffer pointing to the data to checksum.
   * @param checksums the DirectByteBuffer into which checksums will be
   *                  stored. Enough space must be available in this
   *                  buffer to put the checksums.
   */
  public void calculateChunkedSums(ByteBuffer data, ByteBuffer checksums) {
    if (type.size == 0) return;

View on GitHub (pinned to 2add963021)

Solutions

1. Recreate the file from a good source with `hadoop fs -put` so the data and .crc regenerate together
2. If the data file is known-good but the .crc is stale, delete the .crc sidecar and re-copy/rewrite the file
3. For HDFS, run `hdfs fsck` to locate and repair corrupted blocks via re-replication
4. As a lossy last resort for irreplaceable local files, set io.skip.checksum.errors=true to skip bad chunks

Example fix

# before: data copied without its checksum sidecar -> checksum mismatch on read
scp host:/data/part-0000 /dfs/data/
# after: copy through Hadoop so file and .crc stay consistent
hadoop fs -put part-0000 /data/part-0000

try (FSDataInputStream in = fs.open(path)) {
  in.readFully(buf);
} catch (org.apache.hadoop.fs.ChecksumException e) {
  long badPos = e.getPos();
  // data and stored checksums disagree at badPos: re-fetch from a good source or another replica
  restoreFromBackup(path);
}

When it happens

Trigger: Reading a LocalFileSystem (ChecksumFileSystem) file whose .crc sidecar disagrees with the data — file modified outside Hadoop or copied without its .crc; a corrupted HDFS block chunk; bit rot or a partial write leaving data and checksums out of sync.

Common situations: Files rewritten in place by external tools leaving a stale .crc; scp/rsync copying data files without the .crc sidecars; failing disks; interrupted transfers.

Related errors

• Checksum file not a length multiple of checksum size in {} a
• Checksum error: {} at {}
• Checksum file not a length multiple of checksum size in {} a
• Checksum error: {} at {}
• Checksum error: {} at {} exp: {} got: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fb077cd53fe714e9. Report an issue: GitHub.