apache/hadoop · error · HadoopIllegalArgumentException

Invalid checksum type: userOpt=${userOpt}, default=${default

Error message

Invalid checksum type: userOpt=${userOpt}, default=${defaultChecksumOpt}, effective=null

What it means

On the write path the client builds a DataChecksum from the effective checksum option: dfs.checksum.type / dfs.bytes-per-checksum defaults merged with any per-call ChecksumOpt. DataChecksum.newDataChecksum returns null for types that cannot back a writer (NULL, DEFAULT, MIXED, or an unrecognized value), and DfsClientConf.createChecksum converts that null into HadoopIllegalArgumentException.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java:417

  // Construct a checksum option from conf
  public static ChecksumOpt getChecksumOptFromConf(Configuration conf) {
    DataChecksum.Type type = getChecksumType(conf);
    int bytesPerChecksum = conf.getInt(DFS_BYTES_PER_CHECKSUM_KEY,
        DFS_BYTES_PER_CHECKSUM_DEFAULT);
    return new ChecksumOpt(type, bytesPerChecksum);
  }

  /** create a DataChecksum with the given option. */
  public DataChecksum createChecksum(ChecksumOpt userOpt) {
    // Fill in any missing field with the default.
    ChecksumOpt opt = ChecksumOpt.processChecksumOpt(
        defaultChecksumOpt, userOpt);
    DataChecksum dataChecksum = DataChecksum.newDataChecksum(
        opt.getChecksumType(),
        opt.getBytesPerChecksum());
    if (dataChecksum == null) {
      throw new HadoopIllegalArgumentException("Invalid checksum type: userOpt="
          + userOpt + ", default=" + defaultChecksumOpt
          + ", effective=null");
    }
    return dataChecksum;
  }

  @VisibleForTesting
  public int getBlockWriteLocateFollowingInitialDelayMs() {
    return blockWriteLocateFollowingInitialDelayMs;
  }

  public int getBlockWriteLocateFollowingMaxDelayMs() {
    return blockWriteLocateFollowingMaxDelayMs;
  }

  /**
   * @return the hdfsTimeout
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Set dfs.checksum.type to crc32 or crc32c (case-insensitive).
2. Remove the key entirely to fall back to the built-in default.
3. Validate any ChecksumOpt passed to FileSystem.create() — only CRC32/CRC32C can create a client checksummer.
4. Do not set the client-side checksum type to null expecting checksum-free writes; that combination is rejected here by design.

Example fix

# before (hdfs-site.xml)
<property><name>dfs.checksum.type</name><value>md5</value></property>

# after
<property><name>dfs.checksum.type</name><value>crc32c</value></property>

String type = conf.get("dfs.checksum.type", "crc32");
Set<String> valid = Set.of("crc32", "crc32c");
if (!valid.contains(type.toLowerCase(Locale.ROOT))) {
  throw new IllegalArgumentException(
      "dfs.checksum.type must be one of " + valid + ", got: " + type);
}

When it happens

Trigger: FileSystem.create/append with dfs.checksum.type set to an invalid string ('md5', 'none', a typo) or to null/default/mixed; or passing a ChecksumOpt whose type resolves to one of the non-creatable values.

Common situations: Copy-pasted configuration meant to 'disable checksums' — NULL cannot create a client-side checksummer, so checksum-free writes are not configured this way; typos in hdfs-site.xml; code building ChecksumOpt from unvalidated user input.

Related errors

• Invalid values: dfs.bytes-per-checksum (={}) must divide cel
• {key} = {v} <= 0
• Requested replication factor of {replication}{err} for {src}
• dfs.datanode.parallel.volumes.load.threads.num = {} < 1
• Invalid checksum type in dfs.checksum.type: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2542d4666dd3d2fd. Report an issue: GitHub.