What causes "Must specify either create or append" in apache/hadoop?

Use the factory methods that set the flag for you: dfs.createFile(path).overwrite(true).build() or dfs.appendFile(path).build()

apache/hadoop · error · HadoopIllegalArgumentException

Must specify either create or append

Error message

Must specify either create or append

What it means

Thrown by HdfsDataOutputStreamBuilder.build() in DistributedFileSystem when the builder's CreateFlag set contains neither CREATE/OVERWRITE nor APPEND. The builder API is mode-explicit: you must say whether you are creating or appending before build() dispatches to DFSClient.create/createNonRecursive/append. Hitting it almost always means you used the raw builder() entry point instead of createFile()/appendFile(), which pre-set the flags for you.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:3986

      }
      if (getFlags().contains(CreateFlag.CREATE) ||
          getFlags().contains(CreateFlag.OVERWRITE)) {
        if (isRecursive()) {
          return dfs.create(getPath(), getPermission(), getFlags(),
              getBufferSize(), getReplication(), getBlockSize(),
              getProgress(), getChecksumOpt(), getFavoredNodes(),
              getEcPolicyName(), getStoragePolicyName());
        } else {
          return dfs.createNonRecursive(getPath(), getPermission(), getFlags(),
              getBufferSize(), getReplication(), getBlockSize(), getProgress(),
              getChecksumOpt(), getFavoredNodes(), getEcPolicyName(),
              getStoragePolicyName());
        }
      } else if (getFlags().contains(CreateFlag.APPEND)) {
        return dfs.append(getPath(), getFlags(), getBufferSize(), getProgress(),
            getFavoredNodes());
      }
      throw new HadoopIllegalArgumentException(
          "Must specify either create or append");
    }
  }

  /**
   * Create a HdfsDataOutputStreamBuilder to create a file on DFS.
   * Similar to {@link #create(Path)}, file is overwritten by default.
   *
   * @param path the path of the file to create.
   * @return A HdfsDataOutputStreamBuilder for creating a file.
   */
  @Override
  public HdfsDataOutputStreamBuilder createFile(Path path) {
    return new HdfsDataOutputStreamBuilder(this, path).create().overwrite(true);
  }

  /**
   * Returns a RemoteIterator which can be used to list all open files

View on GitHub (pinned to 2add963021)

Solutions

Use the factory methods that set the flag for you: dfs.createFile(path).overwrite(true).build() or dfs.appendFile(path).build()
If using the raw builder, add the mode explicitly: builder().path(p).overwrite(true) (sets OVERWRITE) or set CreateFlag.APPEND for append
Audit shared/generic code that calls FileSystem#builder or #createDataOutputStreamBuilder and ensure a mode is set before build()
If you wrap builders, assert getFlags() intersects {CREATE, OVERWRITE, APPEND} before delegating

Example fix

// before
FSDataOutputStream out = ((DistributedFileSystem) fs).builder()
    .path(new Path("/data/out.txt"))
    .replication((short) 3)
    .build(); // HadoopIllegalArgumentException: Must specify either create or append

// after
FSDataOutputStream out = ((DistributedFileSystem) fs)
    .createFile(new Path("/data/out.txt"))
    .overwrite(true)
    .replication((short) 3)
    .build();

Defensive patterns

Strategy: validation

Validate before calling

// Prefer factory methods that set the mode flag; guard generic builders:
DistributedFileSystem dfs = (DistributedFileSystem) fs;
try (FSDataOutputStream out = dfs.createFile(path).overwrite(true).build()) {
  // write
}
// If you must use the raw builder, always pair it with a mode:
// dfs.builder().path(p).overwrite(true)...  or .append()

Try / catch

try {
  out = builder.build();
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().contains("Must specify either create or append")) {
    throw new IllegalStateException("Builder used without create/append mode", e);
  }
  throw e;
}

Prevention

Never call builder().build() without createFile()/appendFile() or an explicit .overwrite()/.create()/.append()
In shared filesystem utility code, branch on requested mode and delegate to createFile() or appendFile() explicitly
Add a unit test that exercises both modes of any builder wrapper you own

When it happens

Trigger: Calling ((DistributedFileSystem) fs).builder().path(p)...build() (or FileSystem#createDataOutputStreamBuilder) without ever setting a mode flag; constructing HdfsDataOutputStreamBuilder manually and only setting permission/bufferSize/replication; calling .overwrite(false) on a builder that never had CREATE set (OVERWRITE alone counts, but a builder stripped of flags has none).

Common situations: Migrating code from fs.create()/fs.append() to the builder API added in Hadoop 2.8/3.x and forgetting the mode call; copy-pasting a builder chain that omits .overwrite(); generic code that obtains FileSystem.builder() polymorphically and assumes a default mode.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d4bed0bf0ddf6299. Report an issue: GitHub.