apache/hadoop · critical · ChecksumMismatchException

Expected checksum is %s while actual checksum is %s

Error message

Expected checksum is %s while actual checksum is %s

What it means

ObjectContent bundles an object InputStream with the checksum the connector computed while reading (ChainTOSInputStream derives it from TOS response headers such as x-tos-hash-crc64ecma). verifiedStream(expected) compares expected against that computed checksum and, on mismatch, quietly closes the stream and throws ChecksumMismatchException ("Expected checksum is X while actual checksum is Y"), which extends IOException. It signals that the bytes read do not match the checksum recorded when the object metadata was captured.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/ObjectContent.java:43

import java.util.Arrays;

public class ObjectContent {
  private final byte[] checksum;
  private final InputStream stream;

  public ObjectContent(byte[] checksum, InputStream stream) {
    this.checksum = checksum;
    this.stream = stream;
  }

  public InputStream stream() {
    return stream;
  }

  public InputStream verifiedStream(byte[] expectedChecksum) throws ChecksumMismatchException {
    if (!Arrays.equals(expectedChecksum, checksum)) {
      CommonUtils.runQuietly(stream::close);
      throw new ChecksumMismatchException(expectedChecksum, checksum);
    }

    return stream;
  }

  public byte[] checksum() {
    return checksum;
  }
}

View on GitHub (pinned to 2add963021)

Solutions

  1. Re-open and re-read the object once to rule out transient corruption
  2. Pin fs.tos.checksum-type to one value (CRC64ECMA or CRC32C) on every client/job that writes and reads the data
  3. If the object was legitimately rewritten, refresh the expected checksum from a fresh objectStatus instead of a stale one
  4. If the mismatch persists on stable data, treat it as corruption: quarantine the object and restore from a replica/backup

Example fix

// before
try (InputStream in = content.verifiedStream(expected)) {
  consume(in);
}

// after: re-fetch expected from live metadata, retry once
try (InputStream in = content.verifiedStream(expected)) {
  consume(in);
} catch (ChecksumMismatchException e) {
  byte[] fresh = storage.objectStatus(key).checksum();
  try (ObjectContent c = storage.get(key, 0, -1);
       InputStream in = c.verifiedStream(fresh)) {
    consume(in);
  }
}
Defensive patterns

Strategy: retry

Validate before calling

byte[] expected = storage.objectStatus(key).checksum();
try (InputStream in = content.verifiedStream(expected)) {
  consume(in);
}

Try / catch

try (InputStream in = content.verifiedStream(expected)) {
  consume(in);
} catch (ChecksumMismatchException e) {
  // one bounded retry with fresh metadata, then surface as corruption
  byte[] fresh = storage.objectStatus(key).checksum();
  try (ObjectContent c = storage.get(key, 0, -1);
       InputStream in = c.verifiedStream(fresh)) {
    consume(in);
  }
}

Prevention

When it happens

Trigger: Verifying a read against an expected checksum captured earlier while (a) the object was overwritten or truncated by another writer, (b) expected was computed under a different fs.tos.checksum-type (CRC64ECMA default vs CRC32C), or (c) a proxy/HTTP layer corrupted the range payload.

Common situations: fs.tos.checksum-type changed between the writing and reading jobs; concurrent writers mutating the same key; unstable middleboxes altering bodies; mixed connector versions.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a224fe4de40e7823. Report an issue: GitHub.