apache/hadoop · error · HadoopIllegalArgumentException

BytesPerChecksum option is set multiple times

Error message

BytesPerChecksum option is set multiple times

What it means

AbstractFileSystem.create permits CreateOpts.BytesPerChecksum at most once; a second occurrence throws HadoopIllegalArgumentException up front. bytesPerChecksum is the divisor used later for the blockSize divisibility check, so a duplicated value would make validation meaningless.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java:566

          throw new HadoopIllegalArgumentException(
              "BlockSize option is set multiple times");
        }
        blockSize = ((CreateOpts.BlockSize) iOpt).getValue();
      } else if (CreateOpts.BufferSize.class.isInstance(iOpt)) {
        if (bufferSize != -1) {
          throw new HadoopIllegalArgumentException(
              "BufferSize option is set multiple times");
        }
        bufferSize = ((CreateOpts.BufferSize) iOpt).getValue();
      } else if (CreateOpts.ReplicationFactor.class.isInstance(iOpt)) {
        if (replication != -1) {
          throw new HadoopIllegalArgumentException(
              "ReplicationFactor option is set multiple times");
        }
        replication = ((CreateOpts.ReplicationFactor) iOpt).getValue();
      } else if (CreateOpts.BytesPerChecksum.class.isInstance(iOpt)) {
        if (bytesPerChecksum != -1) {
          throw new HadoopIllegalArgumentException(
              "BytesPerChecksum option is set multiple times");
        }
        bytesPerChecksum = ((CreateOpts.BytesPerChecksum) iOpt).getValue();
      } else if (CreateOpts.ChecksumParam.class.isInstance(iOpt)) {
        if (checksumOpt != null) {
          throw new  HadoopIllegalArgumentException(
              "CreateChecksumType option is set multiple times");
        }
        checksumOpt = ((CreateOpts.ChecksumParam) iOpt).getValue();
      } else if (CreateOpts.Perms.class.isInstance(iOpt)) {
        if (permission != null) {
          throw new HadoopIllegalArgumentException(
              "Perms option is set multiple times");
        }
        permission = ((CreateOpts.Perms) iOpt).getValue();
      } else if (CreateOpts.Progress.class.isInstance(iOpt)) {
        if (progress != null) {
          throw new HadoopIllegalArgumentException(

View on GitHub (pinned to 2add963021)

Solutions

  1. Pass CreateOpts.bytesPerChecksum exactly once
  2. Replace any existing occurrence with CreateOpts.setOpt(CreateOpts.bytesPerChecksum(n), opts)
  3. Dedupe the opts array by option class before create()

Example fix

// before
fc.create(f, flag, CreateOpts.bytesPerChecksum(512),
    CreateOpts.bytesPerChecksum(1024));

// after
fc.create(f, flag, CreateOpts.bytesPerChecksum(1024));
Defensive patterns

Strategy: validation

Validate before calling

if (CreateOpts.getOpt(CreateOpts.BytesPerChecksum.class, opts) != null) {
  opts = CreateOpts.setOpt(CreateOpts.bytesPerChecksum(n), opts); // replace, don't append
} else { /* append */ }

Prevention

When it happens

Trigger: Passing CreateOpts.bytesPerChecksum(...) twice, e.g. a checksum-tuning layer appending its value after the caller already set one.

Common situations: Applications optimizing checksum settings merging their option with user options; config keys translated to opts in two places (client default + explicit call); duplicated tuning code after refactoring.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5020c4b423cd3d9f. Report an issue: GitHub.