apache/hadoop · error · HadoopIllegalArgumentException

Invalid checksum type: userOpt={}, default={}, effective=nul

Error message

Invalid checksum type: userOpt={}, default={}, effective=null

What it means

'hdfs debug computeMeta' recomputes a block metadata file and needs a DataChecksum matching the client settings. It reads the effective ChecksumOpt via DfsClientConf.getChecksumOptFromConf(conf) (dfs.checksum.type, dfs.bytes-per-checksum) and calls DataChecksum.newDataChecksum(type, bytesPerChecksum). That factory returns null — instead of throwing — when bytesPerChecksum <= 0 or the type is not creatable (only NULL, CRC32 and CRC32C can construct a checksum; DEFAULT and MIXED are markers). DebugAdmin converts the null into this HadoopIllegalArgumentException. The message text has a known formatting quirk: it prints the same option twice and always says 'effective=null'.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DebugAdmin.java:262

          "  Compute HDFS metadata from the specified block file, and save it"
              + " to" + System.lineSeparator()
              + "  the specified output metadata file."
              + System.lineSeparator() + System.lineSeparator()
              + "**NOTE: Use at your own risk!" + System.lineSeparator()
              + " If the block file is corrupt"
              + " and you overwrite it's meta file, " + System.lineSeparator()
              + " it will show up"
              + " as good in HDFS, but you can't read the data."
              + System.lineSeparator()
              + " Only use as a last measure, and when you are 100% certain"
              + " the block file is good.");
    }

    private DataChecksum createChecksum(Options.ChecksumOpt opt) {
      DataChecksum dataChecksum = DataChecksum
          .newDataChecksum(opt.getChecksumType(), opt.getBytesPerChecksum());
      if (dataChecksum == null) {
        throw new HadoopIllegalArgumentException(
            "Invalid checksum type: userOpt=" + opt + ", default=" + opt
                + ", effective=null");
      }
      return dataChecksum;
    }

    int run(List<String> args) throws IOException {
      if (args.size() == 0) {
        System.out.println(usageText);
        System.out.println(helpText + System.lineSeparator());
        return 1;
      }
      final String name = StringUtils.popOptionWithArgument("-block", args);
      if (name == null) {
        System.err.println("You must specify a block file with -block");
        return 2;
      }
      final File blockFile = new File(name);

View on GitHub (pinned to 2add963021)

Solutions

1. Set dfs.checksum.type to a creatable value: crc32, crc32c, or null (case-insensitive).
2. Ensure dfs.bytes-per-checksum is a positive integer (default 512).
3. Run computeMeta with an explicit clean config: 'hdfs --config <clean-conf-dir> debug computeMeta ...' to bypass polluted settings.
4. Confirm the block file's real checksum type from its .meta header before overwriting metadata, since computeMeta trusts config, not the block.

Example fix

# before: bad checksum settings in the active config
hdfs debug computeMeta -block blk_1073741825 -out /tmp/blk_1073741825.meta
# -> Invalid checksum type ... effective=null

# after: force valid values
hdfs dfsadmin -fs hdfs://nn -setChecksumType crc32c 2>/dev/null; \
  hdfs --config /etc/hadoop-clean debug computeMeta \
  -block blk_1073741825 -out /tmp/blk_1073741825.meta

# or in core-site.xml
<property><name>dfs.checksum.type</name><value>crc32c</value></property>
<property><name>dfs.bytes-per-checksum</name><value>512</value></property>

Configuration conf = new Configuration();
String type = conf.get(DFSConfigKeys.DFS_CHECKSUM_TYPE_KEY, "crc32c");
int bpc = conf.getInt(DFSConfigKeys.DFS_BYTES_PER_CHECKSUM_KEY,
    DFSConfigKeys.DFS_BYTES_PER_CHECKSUM_DEFAULT);
boolean creatable = bpc > 0
    && ("crc32".equalsIgnoreCase(type) || "crc32c".equalsIgnoreCase(type)
        || "null".equalsIgnoreCase(type));
if (!creatable) {
  throw new IllegalArgumentException(
      "dfs.checksum.type=" + type + " / dfs.bytes-per-checksum=" + bpc
          + " cannot build a DataChecksum for computeMeta");
}

try {
  debugAdmin.run(new String[]{"computeMeta", "-block", blk, "-out", out});
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().startsWith("Invalid checksum type")) {
    // fix dfs.checksum.type / dfs.bytes-per-checksum in the active config, then rerun
    System.err.println("Fix dfs.checksum.type (crc32|crc32c|null) and a positive "
        + "dfs.bytes-per-checksum, then retry");
    return;
  }
  throw e;
}

When it happens

Trigger: Running 'hdfs debug computeMeta -block <file> -out <meta>' with dfs.bytes-per-checksum set to 0 or negative, or dfs.checksum.type resolving to a non-creatable type value (DEFAULT/MIXED) in the loaded configuration.

Common situations: A core-site.xml/hdfs-site.xml on the admin box overrides dfs.checksum.type or dfs.bytes-per-checksum with an invalid value for other tooling; debugging on a host with leftover experimental checksum config; NULL checksum expected but typo'd.

Related errors

• Invalid checksum type in dfs.checksum.type: {}
• Internal error: default blockSize is not a multiple of defau
• Can not create a Path from a null string
• Can not create a Path from an empty string
• Not a valid Boolean value for {property} in reconfSlowPeerPa

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6afe0c4f9d523b0a. Report an issue: GitHub.