apache/hadoop · error · AbfsInvalidChecksumException

Checksum Validation Failed, MD5 Mismatch Error, rId: {activi

Error message

Checksum Validation Failed, MD5 Mismatch Error, rId: {activityId}

What it means

Read checksum validation failed: for a ranged read (<= 4 MB with Content-MD5 requested), the MD5 hash returned by the service did not match the hash computed over the bytes the client actually received. AbfsInvalidChecksumException (carrying the request id) means the data was altered or truncated somewhere between Azure Storage and the reading client.

Source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:1421

   */
  protected void verifyCheckSumForRead(final byte[] buffer,
      final AbfsHttpOperation result, final int bufferOffset)
      throws AbfsRestOperationException {
    // Number of bytes returned by server could be less than or equal to what
    // caller requests. In case it is less, extra bytes will be initialized to 0
    // Server returned MD5 Hash will be computed on what server returned.
    // We need to get exact data that server returned and compute its md5 hash
    // Computed hash should be equal to what server returned.
    int numberOfBytesRead = (int) result.getBytesReceived();
    if (numberOfBytesRead == 0) {
      return;
    }
    String md5HashComputed = computeMD5Hash(buffer, bufferOffset,
        numberOfBytesRead);
    String md5HashActual = result.getResponseHeader(CONTENT_MD5);
    if (!md5HashComputed.equals(md5HashActual)) {
      LOG.debug("Md5 Mismatch Error in Read Operation. Server returned Md5: {}, Client computed Md5: {}", md5HashActual, md5HashComputed);
      throw new AbfsInvalidChecksumException(result.getRequestId());
    }
  }

  /**
   * Conditions check for allowing checksum support for read operation.
   * Sending MD5 Hash in request headers. For more details refer to
   * <a href="https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read">Path - Read Azure Storage Rest API</a>.
   * 1. Range header must be present as one of the request headers.
   * 2. buffer length must be less than or equal to 4 MB.
   * @param requestHeaders to be checked for range header.
   * @param rangeHeader must be present.
   * @param bufferLength must be less than or equal to 4 MB.
   * @return true if all conditions are met.
   */
  protected boolean isChecksumValidationEnabled(List<AbfsHttpHeader> requestHeaders,
      final AbfsHttpHeader rangeHeader, final int bufferLength) {
    return getAbfsConfiguration().getIsChecksumValidationEnabled()
        && requestHeaders.contains(rangeHeader) && bufferLength <= 4 * ONE_MB;

View on GitHub (pinned to 2add963021)

Solutions

  1. Retry the read — transient corruption succeeds on a fresh stream.
  2. Bypass or fix intermediaries: exclude the storage endpoint from proxies/TLS inspection and verify no bytes are rewritten.
  3. Check for concurrent writers appending to the file being read; snapshot or finalize files before consumers read them.
  4. If reproducible on one range, cross-check with an independent tool (AzCopy/Storage Explorer) to distinguish in-transit corruption from data-level issues, and report with the activityId.

Example fix

// before
try (FSDataInputStream in = fs.open(path)) { in.readFully(off, buf); }
// after
for (int i = 0; i < 3; i++) {
  try (FSDataInputStream in = fs.open(path)) {
    in.readFully(off, buf);
    break;
  } catch (AbfsInvalidChecksumException e) {
    if (i == 2) throw e;
  }
}
Defensive patterns

Strategy: retry

Try / catch

catch (AbfsInvalidChecksumException ex) {
  // close and reopen the stream, re-read the same range;
  // after N failures escalate — likely proxy corruption or concurrent writes
  retryReadRange(offset, len);
}

Prevention

When it happens

Trigger: A read with checksum validation enabled completes, but bytes in the buffer hash differently than the server-supplied Content-MD5 — corrupted/truncated transfer, a proxy rewriting bytes, or reading a range that was concurrently modified so the served content no longer matches the checksum.

Common situations: Corporate proxies, TLS-inspection appliances, or antivirus mangling bodies; flaky networks truncating reads; readers racing with writers appending to the same file; bugs in custom HTTP layers between Hadoop and the service.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c1b4171f9677e36e. Report an issue: GitHub.