apache/hadoop · error · HadoopIllegalArgumentException

Progress option is set multiple times

Error message

Progress option is set multiple times

What it means

AbstractFileSystem.create accepts CreateOpts.Progress at most once; a second Progressable option in the varargs throws HadoopIllegalArgumentException before the file is created. The progress callback must be unambiguous, hence the strict single-occurrence rule.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java:584

          throw new HadoopIllegalArgumentException(
              "BytesPerChecksum option is set multiple times");
        }
        bytesPerChecksum = ((CreateOpts.BytesPerChecksum) iOpt).getValue();
      } else if (CreateOpts.ChecksumParam.class.isInstance(iOpt)) {
        if (checksumOpt != null) {
          throw new  HadoopIllegalArgumentException(
              "CreateChecksumType option is set multiple times");
        }
        checksumOpt = ((CreateOpts.ChecksumParam) iOpt).getValue();
      } else if (CreateOpts.Perms.class.isInstance(iOpt)) {
        if (permission != null) {
          throw new HadoopIllegalArgumentException(
              "Perms option is set multiple times");
        }
        permission = ((CreateOpts.Perms) iOpt).getValue();
      } else if (CreateOpts.Progress.class.isInstance(iOpt)) {
        if (progress != null) {
          throw new HadoopIllegalArgumentException(
              "Progress option is set multiple times");
        }
        progress = ((CreateOpts.Progress) iOpt).getValue();
      } else if (CreateOpts.CreateParent.class.isInstance(iOpt)) {
        if (createParent != null) {
          throw new HadoopIllegalArgumentException(
              "CreateParent option is set multiple times");
        }
        createParent = ((CreateOpts.CreateParent) iOpt).getValue();
      } else {
        throw new HadoopIllegalArgumentException("Unkown CreateOpts of type " +
            iOpt.getClass().getName());
      }
    }
    if (permission == null) {
      throw new HadoopIllegalArgumentException("no permission supplied");
    }

View on GitHub (pinned to 2add963021)

Solutions

  1. Attach CreateOpts.progress exactly once; drop the redundant one
  2. Use CreateOpts.setOpt(CreateOpts.progress(p), opts) to replace
  3. Dedupe opts by class in shared create wrappers

Example fix

// before
fc.create(f, flag, CreateOpts.progress(cb1), CreateOpts.progress(cb2));

// after
fc.create(f, flag, CreateOpts.progress(cb2));
Defensive patterns

Strategy: validation

Validate before calling

opts = CreateOpts.setOpt(CreateOpts.progress(callback), opts); // replaces any existing Progress option

Prevention

When it happens

Trigger: Two CreateOpts.progress(...) entries in one create() call — e.g. a job framework attaching its own progressable after the application already supplied one.

Common situations: Frameworks (MR/Tez-style runners, connectors) that auto-attach progress reporters layered over application opts via concatenation.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9c5ace60c06bbd90. Report an issue: GitHub.