apache/hadoop · error · IOException

Unknown algorithm

Error message

Unknown algorithm: ${algorithm}

What it means

JsonUtilClient.toMD5MD5CRC32FileChecksum converts the JSON returned by WebHDFS GETFILECHECKSUM into an MD5MD5CRC32FileChecksum. The client-side reconstruction supports only CRC32 and CRC32C checksum families. The default branch throws this IOException when the checksum algorithm advertised by the server resolves to an unsupported type.

Solutions

1. Run the same getFileChecksum operation with a Hadoop client release matching the cluster to confirm the algorithm is supported end to end.
2. Point directly at the cluster WebHDFS endpoint and remove any gateway that rewrites checksum JSON.
3. Upgrade the client or server to versions that both support the checksum type in use.
4. If an unsupported checksum type is expected, compute an application-level digest by reading the file instead of relying on getFileChecksum.

Example fix

// before
FileChecksum checksum = fs.getFileChecksum(path); // algorithm unsupported by client

// after: fall back to reading the bytes when WebHDFS cannot decode the server checksum
try {
  return fs.getFileChecksum(path);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Unknown algorithm")) {
    return computeDigestByReading(fs.open(path));
  }
  throw e;
}

try {
  return fs.getFileChecksum(path);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Unknown algorithm")) {
    return computeDigestByReading(fs.open(path));
  }
  throw e;
}

When it happens

Trigger: Calling FileSystem.getFileChecksum(path) through webhdfs:// or swebhdfs:// when the response's FileChecksum.algorithm does not represent one of the supported CRC32 or CRC32C MD5 composite checksums. This can happen with a nonstandard server, a proxy-modified response, or a Hadoop version whose checksum representation is not understood by this client.

Common situations: A WebHDFS gateway generates checksums in another format; a client is older or newer than the cluster; a test mock returns a generic algorithm string; an intermediary rewrites the JSON checksum object.

Related errors

• Algorithm not matched. Expected
• Invalid value in server response: name=
• Length not matched: length=
• Access denied: dfs.http.policy is HTTPS_ONLY.
• Content-Type " " is incompatible with "application/json"…

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c158d43563f52c3d. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java:562

    final int length = ((Number) m.get("length")).intValue();
    final byte[] bytes = StringUtils.hexStringToByte((String) m.get("bytes"));

    final DataInputStream in = new DataInputStream(
        new ByteArrayInputStream(bytes));
    final DataChecksum.Type crcType =
        MD5MD5CRC32FileChecksum.getCrcTypeFromAlgorithmName(algorithm);
    final MD5MD5CRC32FileChecksum checksum;

    // Recreate what DFSClient would have returned.
    switch(crcType) {
    case CRC32:
      checksum = new MD5MD5CRC32GzipFileChecksum();
      break;
    case CRC32C:
      checksum = new MD5MD5CRC32CastagnoliFileChecksum();
      break;
    default:
      throw new IOException("Unknown algorithm: " + algorithm);
    }
    checksum.readFields(in);

    //check algorithm name
    if (!checksum.getAlgorithmName().equals(algorithm)) {
      throw new IOException("Algorithm not matched. Expected " + algorithm
          + ", Received " + checksum.getAlgorithmName());
    }
    //check length
    if (length != checksum.getLength()) {
      throw new IOException("Length not matched: length=" + length
          + ", checksum.getLength()=" + checksum.getLength());
    }

    return checksum;
  }

  /** Convert a Json map to a AclStatus object. */

View on GitHub (pinned to 2add963021)