apache/hadoop · error · UnsupportedOperationException

non-posix bucket. Append is not supported by OBSFileSystem

Error message

non-posix bucket. Append is not supported by OBSFileSystem

What it means

In OBSFileSystem.create with CreateFlags, if the caller passes CreateFlag.APPEND but the connector was mounted against a plain object (non-POSIX) OBS bucket, it throws UnsupportedOperationException('non-posix bucket. Append is not supported by OBSFileSystem'). Append semantics only exist for OBS POSIX buckets (isFsBucket()); the plain object-storage mode has no server-side append, so the client refuses up front. This is a capability mismatch between the requested operation and the bucket/filesystem mode.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSFileSystem.java:786

   * @throws IOException io exception
   */
  @Override
  @SuppressWarnings("checkstyle:parameternumber")
  public FSDataOutputStream create(
      final Path f,
      final FsPermission permission,
      final EnumSet<CreateFlag> flags,
      final int bufferSize,
      final short replication,
      final long blkSize,
      final Progressable progress,
      final ChecksumOpt checksumOpt)
      throws IOException {
    LOG.debug("create: Creating new file {}, flags:{}, isFsBucket:{}", f,
        flags, isFsBucket());
    if (null != flags && flags.contains(CreateFlag.APPEND)) {
      if (!isFsBucket()) {
        throw new UnsupportedOperationException(
            "non-posix bucket. Append is not supported by "
                + "OBSFileSystem");
      }
      String key = OBSCommonUtils.pathToKey(this, f);
      FileStatus status;
      long objectLen = 0;
      try {
        // get the status or throw an FNFE
        status = getFileStatus(f);
        objectLen = status.getLen();
        // if the thread reaches here, there is something at the path
        if (status.isDirectory()) {
          // path references a directory: automatic error
          throw new FileAlreadyExistsException(f + " is a directory");
        }
      } catch (FileNotFoundException e) {
        LOG.debug("FileNotFoundException, create: Creating new file {}",
            f);

View on GitHub (pinned to 2add963021)

Solutions

  1. Use a bucket with the OBS POSIX (fs) protocol enabled so isFsBucket() returns true, if append is a hard requirement
  2. Otherwise switch the writer to create-with-overwrite or read-modify-rewrite (rewrite the object with appended content) — object storage has no append
  3. Intercept APPEND in application code: detect this UnsupportedOperationException and fall back to non-appending write strategy
  4. Verify fs.obs.* bucket-type configuration matches the actual bucket capability before deploying

Example fix

// before
fs.create(path, FsPermission.getDefault(),
    EnumSet.of(CreateFlag.APPEND), 4096, replication, blockSize, null); // non-POSIX bucket -> UnsupportedOperationException

// after
fs.create(path, FsPermission.getDefault(),
    EnumSet.of(CreateFlag.CREATE, CreateFlag.OVERWRITE), 4096, replication, blockSize, null);
Defensive patterns

Strategy: fallback

Validate before calling

// capability check without touching files: try append on a temp key once at startup
boolean appendSupported;
try {
  Path probe = new Path("/tmp/.append-probe-" + System.nanoTime());
  try (FSDataOutputStream ignored = fs.create(probe, true)) { ignored.write(1); }
  try (FSDataOutputStream ignored = fs.append(probe)) { }
  fs.delete(probe, false);
  appendSupported = true;
} catch (UnsupportedOperationException e) {
  appendSupported = false;
}

Try / catch

try {
  fs.create(f, perm, EnumSet.of(CreateFlag.APPEND), bufSize, repl, blkSize, progress);
} catch (UnsupportedOperationException e) {
  if (String.valueOf(e.getMessage()).contains("Append is not supported")) {
    // fall back to non-appending strategy: CREATE/OVERWRITE or rewrite
    fs.create(f, perm, EnumSet.of(CreateFlag.CREATE, CreateFlag.OVERWRITE), bufSize, repl, blkSize, progress);
  } else {
    throw e;
  }
}

Prevention

When it happens

Trigger: Calling create(..., EnumSet.of(CreateFlag.APPEND), ...) while fs.obs.bucket.type / the filesystem configuration does not indicate a POSIX bucket; running a framework (e.g. HBase WAL writers, some Iceberg/Hive setups) that requests APPEND against a standard object bucket; configs that worked on a POSIX-protocol endpoint pointed later at a plain OBS endpoint.

Common situations: Migrating workloads from HDFS or OBS POSIX buckets to plain OBS object buckets without removing APPEND flags; defaulting an application to append-mode output; provisioning new buckets without the POSIX feature enabled.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a95d8e9fffa735be. Report an issue: GitHub.