apache/hadoop · critical · ChecksumMismatchException

Expected checksum is %s while actual checksum is %s

Error message

Expected checksum is %s while actual checksum is %s

What it means

GetObjectOutput wraps a GetObjectV2Output plus the checksum parsed from the TOS response headers (crc64ecma/crc32c). verifiedContent(expected) compares the two and, on mismatch, force-closes the connection (freeing the socket) and throws ChecksumMismatchException ("Expected checksum is %s while actual checksum is %s"), an IOException. It guards range/random reads against corruption or stale expectations.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/tos/GetObjectOutput.java:51

  public GetObjectOutput(GetObjectV2Output output, byte[] checksum) {
    Preconditions.checkNotNull(checksum, "Checksum should not be null.");
    this.output = output;
    this.checksum = checksum;
  }

  public GetObjectV2Output output() {
    return output;
  }

  public byte[] checksum() {
    return checksum;
  }

  public InputStream verifiedContent(byte[] expectedChecksum) throws IOException {
    if (!Arrays.equals(expectedChecksum, checksum)) {
      CommonUtils.runQuietly(this::forceClose);
      throw new ChecksumMismatchException(expectedChecksum, checksum);
    }

    return output.getContent();
  }

  public void forceClose() throws IOException {
    output.forceClose();
  }
}

View on GitHub (pinned to 2add963021)

Solutions

  1. Re-fetch objectStatus(key) for a fresh expected checksum and retry the read once
  2. Pin fs.tos.checksum-type to one value (default CRC64ECMA) across the cluster
  3. Enforce single-writer per key or version your expectations
  4. If the mismatch is repeatable on stable data, treat it as corruption and restore the object

Example fix

// before
try (InputStream in = getOutput.verifiedContent(expected)) {
  consume(in);
}

// after
try (InputStream in = getOutput.verifiedContent(expected)) {
  consume(in);
} catch (ChecksumMismatchException e) {
  byte[] fresh = storage.objectStatus(key).checksum();
  try (ObjectContent c = storage.get(key, offset, len);
       InputStream in = c.verifiedStream(fresh)) {
    consume(in);
  }
}
Defensive patterns

Strategy: retry

Validate before calling

byte[] expected = storage.objectStatus(key).checksum();
try (InputStream in = getOutput.verifiedContent(expected)) {
  consume(in);
}

Try / catch

try (InputStream in = getOutput.verifiedContent(expected)) {
  consume(in);
} catch (ChecksumMismatchException e) {
  byte[] fresh = storage.objectStatus(key).checksum();
  try (ObjectContent c = storage.get(key, offset, length);
       InputStream in = c.verifiedStream(fresh)) {
    consume(in);
  }
}

Prevention

When it happens

Trigger: Range-read verification where the expected checksum was captured from a different object state: object overwritten between head and GET, expected computed under a different fs.tos.checksum-type (CRC32C vs CRC64ECMA), or the payload corrupted in transit.

Common situations: Concurrent writers replacing the same key; mixed client configurations with different checksum types; flaky proxies altering partial response bodies.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/06f85b538a1f34b0. Report an issue: GitHub.