What causes "BufferSize option is set multiple times" in apache/hadoop?

Pass CreateOpts.bufferSize exactly once — inspect the existing varargs before appending

apache/hadoop · error · HadoopIllegalArgumentException

BufferSize option is set multiple times

Error message

BufferSize option is set multiple times

What it means

AbstractFileSystem.create accepts CreateOpts.BufferSize at most once; a second occurrence in the varargs throws HadoopIllegalArgumentException immediately. This is pure argument validation, raised before the create reaches the underlying file system.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java:554

    int bufferSize = -1;
    short replication = -1;
    long blockSize = -1;
    int bytesPerChecksum = -1;
    ChecksumOpt checksumOpt = null;
    FsPermission permission = null;
    Progressable progress = null;
    Boolean createParent = null;
 
    for (CreateOpts iOpt : opts) {
      if (CreateOpts.BlockSize.class.isInstance(iOpt)) {
        if (blockSize != -1) {
          throw new HadoopIllegalArgumentException(
              "BlockSize option is set multiple times");
        }
        blockSize = ((CreateOpts.BlockSize) iOpt).getValue();
      } else if (CreateOpts.BufferSize.class.isInstance(iOpt)) {
        if (bufferSize != -1) {
          throw new HadoopIllegalArgumentException(
              "BufferSize option is set multiple times");
        }
        bufferSize = ((CreateOpts.BufferSize) iOpt).getValue();
      } else if (CreateOpts.ReplicationFactor.class.isInstance(iOpt)) {
        if (replication != -1) {
          throw new HadoopIllegalArgumentException(
              "ReplicationFactor option is set multiple times");
        }
        replication = ((CreateOpts.ReplicationFactor) iOpt).getValue();
      } else if (CreateOpts.BytesPerChecksum.class.isInstance(iOpt)) {
        if (bytesPerChecksum != -1) {
          throw new HadoopIllegalArgumentException(
              "BytesPerChecksum option is set multiple times");
        }
        bytesPerChecksum = ((CreateOpts.BytesPerChecksum) iOpt).getValue();
      } else if (CreateOpts.ChecksumParam.class.isInstance(iOpt)) {
        if (checksumOpt != null) {
          throw new  HadoopIllegalArgumentException(

View on GitHub (pinned to 2add963021)

Solutions

Pass CreateOpts.bufferSize exactly once — inspect the existing varargs before appending
Replace instead of append with CreateOpts.setOpt(CreateOpts.bufferSize(n), opts)
Dedupe opts by option class before create()

Example fix

// before
fc.create(f, flag, CreateOpts.bufferSize(4096), CreateOpts.bufferSize(8192));

// after
fc.create(f, flag, CreateOpts.bufferSize(8192));

Defensive patterns

Strategy: validation

Validate before calling

if (Collections.frequency(Arrays.asList(opts).stream().map(o -> o.getClass()).collect(toList()), Options.CreateOpts.BufferSize.class) > 1) throw new IllegalArgumentException("bufferSize duplicated");
// or simply: opts = CreateOpts.setOpt(CreateOpts.bufferSize(n), opts);

Prevention

Pass each create option exactly once from a single layer of code
Use CreateOpts.setOpt for overrides
Dedupe opts by class in shared create wrappers

When it happens

Trigger: Passing CreateOpts.bufferSize(...) twice in a single create() call, e.g. a framework defaulting the buffer size being appended after the caller's own buffer size option.

Common situations: Option-merging code that concatenates default opts with user opts; copy-pasted option lists where a buffer size was already present; config-driven option assembly applied twice.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4d10e1008b7c2162. Report an issue: GitHub.