apache/hadoop · error · IOException

Unknown checksum type in ${algorithm}

Error message

Unknown checksum type in ${algorithm}

What it means

MD5MD5CRC32FileChecksum.getCrcTypeFromAlgorithmName() maps a checksum algorithm string (e.g. 'MD5-of-0MD5-of-512CRC32') to a DataChecksum.Type by suffix matching. Only CRC32 and CRC32C suffixes are recognized; any other algorithm name throws IOException.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/MD5MD5CRC32FileChecksum.java:74

    this.crcPerBlock = crcPerBlock;
    this.md5 = md5;
  }

  @Override
  public String getAlgorithmName() {
    return "MD5-of-" + crcPerBlock + "MD5-of-" + bytesPerCRC +
        getCrcType().name();
  }

  public static DataChecksum.Type getCrcTypeFromAlgorithmName(String algorithm)
      throws IOException {
    if (algorithm.endsWith(DataChecksum.Type.CRC32.name())) {
      return DataChecksum.Type.CRC32;
    } else if (algorithm.endsWith(DataChecksum.Type.CRC32C.name())) {
      return DataChecksum.Type.CRC32C;
    }

    throw new IOException("Unknown checksum type in " + algorithm);
  }

  @Override
  public int getLength() {return LENGTH;}

  @Override
  public byte[] getBytes() {
    return WritableUtils.toByteArray(this);
  }

  /**
   * returns the CRC type.
   * @return data check sum type.
   */
  public DataChecksum.Type getCrcType() {
    // default to the one that is understood by all releases.
    return DataChecksum.Type.CRC32;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Align client and cluster Hadoop versions so both sides know the checksum type
2. Regenerate/recompute the checksum with CRC32 or CRC32C and rewrite the sidecar metadata
3. Pre-validate the algorithm string (endsWith CRC32/CRC32C) and branch to an explicit 'unsupported checksum' path

Example fix

// before
DataChecksum.Type t = MD5MD5CRC32FileChecksum.getCrcTypeFromAlgorithmName(alg);
// after
if (!alg.endsWith(DataChecksum.Type.CRC32.name()) && !alg.endsWith(DataChecksum.Type.CRC32C.name())) {
  throw new IOException("Unsupported checksum algorithm: " + alg);
}
DataChecksum.Type t = MD5MD5CRC32FileChecksum.getCrcTypeFromAlgorithmName(alg);

if (algorithm == null
    || (!algorithm.endsWith(DataChecksum.Type.CRC32.name())
        && !algorithm.endsWith(DataChecksum.Type.CRC32C.name()))) {
  throw new IOException("Unsupported checksum algorithm: " + algorithm);
}

try {
  DataChecksum.Type t = MD5MD5CRC32FileChecksum.getCrcTypeFromAlgorithmName(alg);
} catch (IOException e) {
  /* unknown checksum type: skip comparison or upgrade for compatibility */
}

When it happens

Trigger: Feeding an algorithm name that does not end in CRC32 or CRC32C — typically parsed from a file checksum header written by a different/newer Hadoop (new checksum type), a malformed/hand-built algorithm string, or corrupted checksum metadata.

Common situations: Version skew between writer and reader clusters, mixed-version rolling upgrades, custom or experimental checksum types, truncated algorithm strings.

Related errors

• Checksum file not a length multiple of checksum size in {} a
• Checksum ${type} not matched for file ${filename} at positio
• Unexpected blockChecksumType '%s', expecting COMPOSITE_CRC
• key + ": No such file or directory."
• no such method

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9ac1fbc1ede1f05e. Report an issue: GitHub.