What causes "CreateChecksumType option is set multiple times" in apache/hadoop?

Pass CreateOpts.checksumParam(...) once only

apache/hadoop · error · HadoopIllegalArgumentException

CreateChecksumType option is set multiple times

Error message

CreateChecksumType option is set multiple times

What it means

AbstractFileSystem.create accepts CreateOpts.ChecksumParam at most once; a second occurrence throws HadoopIllegalArgumentException (the message text says "CreateChecksumType", the legacy option name). ChecksumParam carries the checksum type/bytes pair, so duplicates are rejected before the merged checksum option is computed.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java:572

          throw new HadoopIllegalArgumentException(
              "BufferSize option is set multiple times");
        }
        bufferSize = ((CreateOpts.BufferSize) iOpt).getValue();
      } else if (CreateOpts.ReplicationFactor.class.isInstance(iOpt)) {
        if (replication != -1) {
          throw new HadoopIllegalArgumentException(
              "ReplicationFactor option is set multiple times");
        }
        replication = ((CreateOpts.ReplicationFactor) iOpt).getValue();
      } else if (CreateOpts.BytesPerChecksum.class.isInstance(iOpt)) {
        if (bytesPerChecksum != -1) {
          throw new HadoopIllegalArgumentException(
              "BytesPerChecksum option is set multiple times");
        }
        bytesPerChecksum = ((CreateOpts.BytesPerChecksum) iOpt).getValue();
      } else if (CreateOpts.ChecksumParam.class.isInstance(iOpt)) {
        if (checksumOpt != null) {
          throw new  HadoopIllegalArgumentException(
              "CreateChecksumType option is set multiple times");
        }
        checksumOpt = ((CreateOpts.ChecksumParam) iOpt).getValue();
      } else if (CreateOpts.Perms.class.isInstance(iOpt)) {
        if (permission != null) {
          throw new HadoopIllegalArgumentException(
              "Perms option is set multiple times");
        }
        permission = ((CreateOpts.Perms) iOpt).getValue();
      } else if (CreateOpts.Progress.class.isInstance(iOpt)) {
        if (progress != null) {
          throw new HadoopIllegalArgumentException(
              "Progress option is set multiple times");
        }
        progress = ((CreateOpts.Progress) iOpt).getValue();
      } else if (CreateOpts.CreateParent.class.isInstance(iOpt)) {
        if (createParent != null) {
          throw new HadoopIllegalArgumentException(

View on GitHub (pinned to 2add963021)

Solutions

Pass CreateOpts.checksumParam(...) once only
Use CreateOpts.setOpt(CreateOpts.checksumParam(opt), opts) to replace an existing one
Dedupe opts by class in shared wrappers

Example fix

// before
fc.create(f, flag, CreateOpts.checksumParam(new ChecksumOpt(ChecksumType.CRC32C, 512)),
    CreateOpts.checksumParam(new ChecksumOpt(ChecksumType.CRC32, 512)));

// after
fc.create(f, flag, CreateOpts.checksumParam(new ChecksumOpt(ChecksumType.CRC32C, 512)));

Defensive patterns

Strategy: validation

Validate before calling

opts = CreateOpts.setOpt(CreateOpts.checksumParam(cOpt), opts); // replaces existing ChecksumParam

Prevention

Set the checksum type/bytes via a single ChecksumParam option
Use CreateOpts.setOpt instead of appending when overriding
Dedupe opts by class in shared create wrappers

When it happens

Trigger: Two CreateOpts.checksumParam(...) entries in one create() call, typically a caller's tuning option plus a framework default appended afterwards.

Common situations: Checksum-type tuning (CRC32 vs CRC32C) applied both by config-derived code and by the caller; option merge helpers that append instead of replace; legacy code still naming the option "CreateChecksumType".

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/02da0249b2d691e8. Report an issue: GitHub.