What causes "BlockSize option is set multiple times" in apache/hadoop?

Remove the duplicate at the call site so blockSize is passed exactly once

apache/hadoop · error · HadoopIllegalArgumentException

BlockSize option is set multiple times

Error message

BlockSize option is set multiple times

What it means

AbstractFileSystem.create scans the CreateOpts varargs once; the second CreateOpts.BlockSize it encounters throws HadoopIllegalArgumentException before any I/O happens. The API accepts each option type at most once so the resolved value is unambiguous.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java:548

  public final FSDataOutputStream create(final Path f,
      final EnumSet<CreateFlag> createFlag, Options.CreateOpts... opts)
      throws AccessControlException, FileAlreadyExistsException,
      FileNotFoundException, ParentNotDirectoryException,
      UnsupportedFileSystemException, UnresolvedLinkException, IOException {
    checkPath(f);
    int bufferSize = -1;
    short replication = -1;
    long blockSize = -1;
    int bytesPerChecksum = -1;
    ChecksumOpt checksumOpt = null;
    FsPermission permission = null;
    Progressable progress = null;
    Boolean createParent = null;
 
    for (CreateOpts iOpt : opts) {
      if (CreateOpts.BlockSize.class.isInstance(iOpt)) {
        if (blockSize != -1) {
          throw new HadoopIllegalArgumentException(
              "BlockSize option is set multiple times");
        }
        blockSize = ((CreateOpts.BlockSize) iOpt).getValue();
      } else if (CreateOpts.BufferSize.class.isInstance(iOpt)) {
        if (bufferSize != -1) {
          throw new HadoopIllegalArgumentException(
              "BufferSize option is set multiple times");
        }
        bufferSize = ((CreateOpts.BufferSize) iOpt).getValue();
      } else if (CreateOpts.ReplicationFactor.class.isInstance(iOpt)) {
        if (replication != -1) {
          throw new HadoopIllegalArgumentException(
              "ReplicationFactor option is set multiple times");
        }
        replication = ((CreateOpts.ReplicationFactor) iOpt).getValue();
      } else if (CreateOpts.BytesPerChecksum.class.isInstance(iOpt)) {
        if (bytesPerChecksum != -1) {
          throw new HadoopIllegalArgumentException(

View on GitHub (pinned to 2add963021)

Solutions

Remove the duplicate at the call site so blockSize is passed exactly once
Use CreateOpts.setOpt(CreateOpts.blockSize(n), opts), which replaces an existing occurrence instead of appending
Run opts through a dedupe that keeps one entry per option class before calling create

Example fix

// before
opts = Arrays.copyOf(opts, opts.length + 1);
opts[opts.length - 1] = CreateOpts.blockSize(bs); // caller may already have passed one
fc.create(f, flag, opts);

// after
opts = CreateOpts.setOpt(CreateOpts.blockSize(bs), opts); // replaces, never duplicates
fc.create(f, flag, opts);

Defensive patterns

Strategy: validation

Validate before calling

static Options.CreateOpts[] dedupe(Options.CreateOpts... opts) {
  Map<Class<?>, Options.CreateOpts> m = new LinkedHashMap<>();
  for (Options.CreateOpts o : opts) m.put(o.getClass(), o); // last occurrence wins
  return m.values().toArray(new Options.CreateOpts[0]);
}
// fc.create(f, flag, dedupe(opts));

Prevention

In wrappers, always use CreateOpts.setOpt(opt, opts) which replaces instead of appending
Never append a default option to an array that may already contain it
Unit-test option-merging helpers with inputs that already carry every option

When it happens

Trigger: Passing CreateOpts.blockSize(...) twice in one create() call — classically a wrapper that appends its default blockSize after forwarding the caller's options array verbatim (Arrays.copyOf + append).

Common situations: Helper/utility create methods that add defaults on top of user opts; merging config-derived opts with caller opts by concatenation; refactors that append an option instead of replacing it.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e253e7965497da05. Report an issue: GitHub.