apache/hadoop · error · IOException

Algorithm not matched. Expected ${algorithm}, Received ${che

Error message

Algorithm not matched. Expected ${algorithm}, Received ${checksum.getAlgorithmName()}

What it means

After selecting a CRC32 or CRC32C checksum class, JsonUtilClient reads the bytes field into that object and then compares the reconstructed algorithm name with the algorithm string sent by the server. The composite name includes crcPerBlock, bytesPerCRC, and CRC type, so this IOException means the encoded checksum bytes and advertised algorithm describe different parameters.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java:568

        MD5MD5CRC32FileChecksum.getCrcTypeFromAlgorithmName(algorithm);
    final MD5MD5CRC32FileChecksum checksum;

    // Recreate what DFSClient would have returned.
    switch(crcType) {
    case CRC32:
      checksum = new MD5MD5CRC32GzipFileChecksum();
      break;
    case CRC32C:
      checksum = new MD5MD5CRC32CastagnoliFileChecksum();
      break;
    default:
      throw new IOException("Unknown algorithm: " + algorithm);
    }
    checksum.readFields(in);

    //check algorithm name
    if (!checksum.getAlgorithmName().equals(algorithm)) {
      throw new IOException("Algorithm not matched. Expected " + algorithm
          + ", Received " + checksum.getAlgorithmName());
    }
    //check length
    if (length != checksum.getLength()) {
      throw new IOException("Length not matched: length=" + length
          + ", checksum.getLength()=" + checksum.getLength());
    }

    return checksum;
  }

  /** Convert a Json map to a AclStatus object. */
  static AclStatus toAclStatus(final Map<?, ?> json) {
    if (json == null) {
      return null;
    }

    final Map<?, ?> m = (Map<?, ?>) json.get(AclStatus.class.getSimpleName());

View on GitHub (pinned to 2add963021)

Solutions

Call the NameNode WebHDFS endpoint directly with curl or a matching Hadoop client and inspect the complete FileChecksum JSON.
Disable or fix checksum-response rewriting in reverse proxies, caches, and test gateways.
Use matching Hadoop client and cluster versions.
Treat the response as invalid rather than retrying the same request, and compute a digest by reading the file if an application checksum is required.

Example fix

// before: server sends algorithm=MD5-of-0MD5-of-512CRC32
// but bytes encode MD5-of-0MD5-of-512CRC32C
FileChecksum cs = fs.getFileChecksum(path);

// after: use the cluster NameNode directly so algorithm and bytes match
FileSystem fs = FileSystem.get(new URI("webhdfs://namenode:9870"), conf);
FileChecksum cs = fs.getFileChecksum(path);

Defensive patterns

Strategy: try-catch

Try / catch

try {
  return fs.getFileChecksum(path);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Algorithm not matched")) {
    throw new IOException("Server checksum response is inconsistent; verify gateway/server compatibility for " + path, e);
  }
  throw e;
}

Prevention

Never modify the algorithm field independently of the bytes field in WebHDFS fixtures or gateways.
Compare checksums only between clients and servers from compatible Hadoop versions.
Log the raw FileChecksum JSON when consistency failures occur so the mismatched field is visible.

When it happens

Trigger: getFileChecksum through WebHDFS returns a FileChecksum JSON object whose algorithm string does not equal the algorithm reconstructed from its bytes. This is a consistency failure in the response, typically caused by a proxy or mock changing one field independently of the other, or by an incompatible server/client serialization.

Common situations: A gateway caches or rewrites the algorithm field; a hand-built JSON response combines values from different files or Hadoop versions; response corruption through an HTTP intermediary; version drift between WebHDFS client and NameNode.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b3a670d462977198. Report an issue: GitHub.