apache/hadoop · error · UnsupportedOperationException

Append is not supported by S3AFileSystem

Error message

Append is not supported by S3AFileSystem

What it means

S3AFileSystem.append() unconditionally throws UnsupportedOperationException. S3 objects are immutable, so appending to an existing object cannot be implemented; rather than hide a slow full read-modify-write behind the append API, hadoop-aws rejects the optional FileSystem append contract outright.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2370

        .bufferSize(bufferSize)
        .must(FS_S3A_CREATE_PERFORMANCE,
            getPerformanceFlags().enabled(PerformanceFlagEnum.Create));
    if (progress != null) {
      builder.progress(progress);
    }
    return builder.build();
  }

  /**
   * Append to an existing file (optional operation).
   * @param f the existing file to be appended.
   * @param bufferSize the size of the buffer to be used.
   * @param progress for reporting progress if it is not null.
   * @throws IOException indicating that append is not supported.
   */
  public FSDataOutputStream append(Path f, int bufferSize,
      Progressable progress) throws IOException {
    throw new UnsupportedOperationException("Append is not supported "
        + "by S3AFileSystem");
  }


  /**
   * Renames Path src to Path dst.  Can take place on local fs
   * or remote DFS.
   *
   * Warning: S3 does not support renames. This method does a copy which can
   * take S3 some time to execute with large files and directories. Since
   * there is no Progressable passed in, this can time out jobs.
   *
   * Note: This implementation differs with other S3 drivers. Specifically:
   * <pre>
   *       Fails if src is a file and dst is a directory.
   *       Fails if src is a directory and dst is a file.
   *       Fails if the parent of dst does not exist or is a file.
   *       Fails if dst is a directory that is not empty.

View on GitHub (pinned to 2add963021)

Solutions

  1. Restructure to write new objects per flush (suffixed or versioned names) and have readers pick the latest
  2. Use HDFS or another mutable store for append-heavy workloads
  3. If unavoidable, implement the read-modify-write yourself: copy the object plus new bytes to a new key, then swap - accepting the cost

Example fix

// before
try (FSDataOutputStream out = fs.append(logPath, 4096)) {
  out.writeBytes(line);
}

// after: S3A has no append - write a new object
Path next = new Path(logPath.getParent(),
    logPath.getName() + "." + System.currentTimeMillis());
try (FSDataOutputStream out = fs.create(next, false)) {
  out.writeBytes(line);
}
Defensive patterns

Strategy: fallback

Validate before calling

static boolean canAppend(FileSystem fs) {
  String scheme = fs.getUri().getScheme();
  return scheme == null || !(scheme.equals("s3a") || scheme.equals("s3n"));
}

Try / catch

Catch UnsupportedOperationException around append() and switch to the new-object write strategy; check fs.getUri().getScheme() first when the code must run on multiple filesystems.

Prevention

When it happens

Trigger: Any fs.append(path) or fs.append(path, bufferSize, progress) call on an s3a:// path; frameworks that probe append support by calling it.

Common situations: Log appenders, streaming sinks and HDFS-era job code ported to S3A; shared libraries that call append when a FileSystem does not advertise the capability.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1339b429be4768f8. Report an issue: GitHub.