apache/hadoop · critical · CompletionException

Checksum error: {} at {} exp: {} got: {}

Error message

Checksum error: {} at {} exp: {} got: {}

What it means

This is ChecksumFileSystem's ByteBuffer read path verifying data chunk by chunk: it CRC32s each bytesPerChecksum (default 512) slice of the returned buffer and compares against the expected value read from the .<name>.crc file. A mismatch throws CompletionException wrapping ChecksumException with the file, the exact failing offset, and both expected and computed CRCs. Detected here means the local data actually differs from what was written - real corruption (bad disk, bit rot, bad copy), not a Hadoop bug.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java:431

          // During last chunk, there may be less than chunk size
          // data preset, so setting the limit accordingly.
          int lastIncompleteChunk = data.remaining() % bytesPerSum;
          current.limit((c * bytesPerSum) + lastIncompleteChunk);
        } else {
          // set the buffer limit to end of every chunk.
          current.limit((c + 1) * bytesPerSum);
        }

        // compute the crc
        crc.reset();
        crc.update(current);
        int expected = sums.get();
        int calculated = (int) crc.getValue();

        if (calculated != expected) {
          // cast of c added to silence findbugs
          long errPosn = dataOffset + (long) c * bytesPerSum;
          throw new CompletionException(new ChecksumException(
              "Checksum error: " + file + " at " + errPosn +
                  " exp: " + expected + " got: " + calculated, errPosn));
        }
      }
      // if everything matches, we return the data
      return data;
    }

    /**
     * Turn off range merging to make buffer recycling more likely (but not guaranteed).
     * @return 0, always
     */
    @Override
    public int maxReadSizeForVectorReads() {
      return S_0;
    }

    /**

View on GitHub (pinned to 2add963021)

Solutions

1. Treat it as data corruption: restore the file (and its .crc) from the authoritative source or backup - retrying the same read will keep failing.
2. Verify the media: check dmesk/smartctl for disk errors and re-copy the file, then confirm with md5sum against the source.
3. If integrity is already assured by other means and you accept the risk, bypass with fs.setVerifyChecksum(false) or fs.file.impl=RawLocalFileSystem.
4. Delete the .crc only if you have confirmed the data is good (e.g. md5 matches source); otherwise you are just hiding corruption.

Example fix

// before
try (FSDataInputStream in = localFs.open(f)) {
  ByteBuffer b = in.read(pool, 65536, EnumSet.noneOf(ReadOption.class));
}

// after: surface corruption and recover from source
try (FSDataInputStream in = localFs.open(f)) {
  ByteBuffer b = in.read(pool, 65536, EnumSet.noneOf(ReadOption.class));
} catch (CompletionException e) {
  if (e.getCause() instanceof ChecksumException) { /* re-fetch file from source */ }
  else throw e;
}

try {
  ByteBuffer b = in.read(pool, 65536, EnumSet.noneOf(ReadOption.class));
} catch (CompletionException ce) {
  Throwable cause = ce.getCause();
  if (cause instanceof ChecksumException) {
    // real local corruption: re-fetch the file from the source, do not retry this stream
    restoreFromSource(path);
  } else {
    throw ce;
  }
}

When it happens

Trigger: Reading a local file through LocalFileSystem (or another ChecksumFileSystem) whose data block no longer matches its .crc: failing disk sectors, silent corruption on NFS/network mounts, a file copied without its crc and paired with a stale crc from another file, or partial writes from a crashed job.

Common situations: Jobs reading local staging/spill data after hardware errors; directories synced with tools that updated the data file but preserved an old .crc; long-lived local caches on unreliable disks.

Related errors

• Checksum file not a length multiple of checksum size in {} a
• Checksum error: {} at {}
• Checksum file not a length multiple of checksum size in {} a
• Checksum error: {} at {}
• Append is not supported by ChecksumFileSystem

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7c0552fcdd573b1a. Report an issue: GitHub.