apache/hadoop · critical · ChecksumException

Checksum error: {} at {}

Error message

Checksum error: {} at {}

What it means

ChecksumFs's reader found data bytes past the end of checksum coverage (eof && nread > 0): the .crc sidecar ended before the data file did, so bytes exist that have no checksum to verify against. It surfaces as ChecksumException with the file and byte position. The usual root cause is that the data file grew (raw append) or its checksum file shrank/aged, i.e., the two are out of sync.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFs.java:264

        int sumLenRead = sums.read(checksum, 0, CHECKSUM_SIZE * checksumsToRead);
        if (sumLenRead >= 0 && sumLenRead % CHECKSUM_SIZE != 0) {
          throw new EOFException("Checksum file not a length multiple of checksum size " +
                                 "in " + file + " at " + pos + " checksumpos: " + checksumPos +
                                 " sumLenread: " + sumLenRead );
        }
        if (sumLenRead <= 0) { // we're at the end of the file
          eof = true;
        } else {
          // Adjust amount of data to read based on how many checksum chunks we read
          len = Math.min(len, bytesPerSum * (sumLenRead / CHECKSUM_SIZE));
        }
      }
      if (pos != datas.getPos()) {
        datas.seek(pos);
      }
      int nread = readFully(datas, buf, offset, len);
      if (eof && nread > 0) {
        throw new ChecksumException("Checksum error: "+file+" at "+pos, pos);
      }
      return nread;
    }
    
    /* Return the file length */
    private long getFileLength() throws IOException, UnresolvedLinkException {
      if (fileLen==-1L) {
        fileLen = fs.getFileStatus(file).getLen();
      }
      return fileLen;
    }
    
    /**
     * Skips over and discards <code>n</code> bytes of data from the
     * input stream.
     *
     * The <code>skip</code> method skips over some smaller number of bytes
     * when reaching end of file before <code>n</code> bytes have been skipped.

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm the diagnosis: fc.setVerifyChecksum(false) and read the tail - if it succeeds the data survived and only checksum coverage is missing.
2. Delete the stale .crc sidecar (hidden dot-file, same dir: /p/.f.crc) so reads skip verification until the next create rebuilds it.
3. If actual corruption is suspected (reads still fail unverified), restore from a replica/backup or recopy the source with hadoop fs -put so a fresh .crc is generated.
4. Stop appending through the raw filesystem, or always delete the .crc after such writes, to prevent recurrence.

Example fix

// before
FSDataInputStream in = fc.open(path);
in.read(buffer); // ChecksumException: Checksum error: /data/f at 65536

// after
fc.setVerifyChecksum(false);
FSDataInputStream in = fc.open(path);
in.readFully(buffer); // succeeds if only the .crc is stale
java.nio.file.Files.deleteIfExists(java.nio.file.Paths.get("/data/.f.crc"));

fc.setVerifyChecksum(false); // read-side bypass when .crc coverage is known-stale
// long dataLen = fc.open(path).getPos(); // stream length still works unverified

try {
  readVerified(fc, path);
} catch (ChecksumException e) {
  // triage: fc.setVerifyChecksum(false); delete stale .crc; restore from replica if data truly bad
}

When it happens

Trigger: Reading a file through FileContext on file:// after the data file was modified outside the checksum layer: appended via RawLocalFileSystem while the .crc stayed stale, .crc deleted from a middle-generation write, or the pair restored/copy-split inconsistently. The exception is thrown from read() when pos reaches the uncovered region.

Common situations: Bypassing ChecksumFs's 'append/truncate not supported' by writing through the raw fs and forgetting to remove the .crc; mixing hadoop fs -put with plain cp/rsync into the same directory; bit-rot on the volume corrupting the .crc itself; changing io.bytes.per.checksum between write and read of the same file pair.

Related errors

• Checksum file not a length multiple of checksum size in {} a
• Checksum error: {} at {}
• Checksum error: {} at {} exp: {} got: {}
• Checksum file not a length multiple of checksum size in {} a
• Checksum ${type} not matched for file ${filename} at positio

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/eae769b2dbdd5e3c. Report an issue: GitHub.