apache/hadoop · critical · ChecksumException

Checksum error: {} at {}

Error message

Checksum error: {} at {}

What it means

In ChecksumFileSystem's read path, `eof` is set when the .crc stream yields no more checksum records; if data bytes were still successfully read from the data file at that position (nread > 0), it throws ChecksumException "Checksum error: <file> at <pos>". Concretely: the data file is longer than its checksum coverage - the .<name>.crc is truncated, stale (file was appended after the crc was written), or belongs to a different version of the file.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java:316

          throw new ChecksumException(
            "Checksum file not a length multiple of checksum size " +
            "in " + file + " at " + pos + " checksumpos: " + checksumPos +
            " sumLenread: " + sumLenRead,
            pos);
        }
        if (sumLenRead <= 0) { // we're at the end of the file
          eof = true;
        } else {
          // Adjust amount of data to read based on how many checksum chunks we read
          len = Math.min(len, bytesPerSum * (sumLenRead / CHECKSUM_SIZE));
        }
      }
      if(pos != datas.getPos()) {
        datas.seek(pos);
      }
      int nread = readFully(datas, buf, offset, len);
      if (eof && nread > 0) {
        throw new ChecksumException("Checksum error: "+file+" at "+pos, pos);
      }
      return nread;
    }

    /**
     * Get the IO Statistics of the nested stream, falling back to
     * null if the stream does not implement the interface
     * {@link IOStatisticsSource}.
     * @return an IOStatistics instance or null
     */
    @Override
    public IOStatistics getIOStatistics() {
      return IOStatisticsSupport.retrieveIOStatistics(datas);
    }

    public static long findChecksumOffset(long dataOffset,
                                          int bytesPerSum) {
      return HEADER_LENGTH + (dataOffset/bytesPerSum) * FSInputChecker.CHECKSUM_SIZE;

View on GitHub (pinned to 2add963021)

Solutions

1. Delete the stale .<name>.crc so the file is read without checksum coverage, or regenerate it by recopying the file through LocalFileSystem.
2. If the data must be trustworthy, restore both file and .crc from the authoritative source instead of bypassing.
3. For an immediate read, disable verification (fs.setVerifyChecksum(false) / `hadoop fs -cat -ignoreCrc`).
4. Fix the writer: only write/append local files through LocalFileSystem APIs so the .crc stays consistent.

Example fix

// before
try (FSDataInputStream in = localFs.open(f)) { IOUtils.copyBytes(in, out, 4096); }

// after: tolerate a stale .crc for this read
localFs.setVerifyChecksum(false);
try (FSDataInputStream in = localFs.open(f)) { IOUtils.copyBytes(in, out, 4096); }
localFs.setVerifyChecksum(true);

long dataLen = localFs.getFileStatus(f).getLen();
long covered = ((localFs.getFileStatus(crcOf(f)).getLen() - 8) / 4L) * bytesPerChecksum;
if (dataLen > covered) {
  // data extends beyond checksum coverage: drop the stale crc or re-copy the file
}

try {
  IOUtils.copyBytes(localFs.open(f), out, 4096, true);
} catch (ChecksumException e) {
  localFs.setVerifyChecksum(false); // accept unverified read for this file only
  IOUtils.copyBytes(localFs.open(f), out, 4096, true);
  localFs.setVerifyChecksum(true);
}

When it happens

Trigger: Reading past the original end of a locally-written file that was later appended by a process that did not update the .crc (plain java/OS appends, text editors); a truncated .crc that happened to end on a 4-byte boundary (so error 496 does not fire but coverage ends early); mixing files whose crc was written with different bytesPerChecksum.

Common situations: Log or output files on local staging are appended by non-Hadoop tooling while Hadoop clients later read them; out-of-band file modification between the Hadoop write and the read.

Related errors

• Checksum file not a length multiple of checksum size in {} a
• Checksum error: {} at {} exp: {} got: {}
• Checksum file not a length multiple of checksum size in {} a
• Checksum error: {} at {}
• Append is not supported by ChecksumFileSystem

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ffb2912b21e0456f. Report an issue: GitHub.