apache/hadoop · error · IOException

Append is not supported!

Error message

Append is not supported!

What it means

AliyunOSSFileSystem.append() unconditionally throws IOException("Append is not supported!"). Object stores have no server-side append: OSS objects are immutable once written, so the Hadoop append contract cannot be implemented. The connector fails fast for any code path that requests an append-mode output stream.

Source

Thrown at hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java:100

  private int blockOutputActiveBlocks;
  private AliyunOSSFileSystemStore store;
  private int maxKeys;
  private int maxReadAheadPartNumber;
  private int maxConcurrentCopyTasksPerDir;
  private ExecutorService boundedThreadPool;
  private ExecutorService boundedCopyThreadPool;

  private static final PathFilter DEFAULT_FILTER = new PathFilter() {
    @Override
    public boolean accept(Path file) {
      return true;
    }
  };

  @Override
  public FSDataOutputStream append(Path path, int bufferSize,
      Progressable progress) throws IOException {
    throw new IOException("Append is not supported!");
  }

  @Override
  public void close() throws IOException {
    try {
      store.close();
      boundedThreadPool.shutdown();
      boundedCopyThreadPool.shutdown();
    } finally {
      super.close();
    }
  }

  @Override
  public FSDataOutputStream create(Path path, FsPermission permission,
      boolean overwrite, int bufferSize, short replication, long blockSize,
      Progressable progress) throws IOException {
    String key = pathToKey(path);

View on GitHub (pinned to 2add963021)

Solutions

  1. Write to a new file (create with overwrite as appropriate) instead of appending; design outputs as new objects per write
  2. For Hive, rely on ACID/new-file semantics so INSERT INTO creates new files rather than appending
  3. If you truly need append semantics, use HDFS or a filesystem that supports it, or buffer externally and rewrite the object atomically

Example fix

// before
FSDataOutputStream out = fs.append(path); // throws

// after
FSDataOutputStream out = fs.create(path, /*overwrite*/ true);
Defensive patterns

Strategy: validation

Validate before calling

// probe once at startup
if (fs instanceof AliyunOSSFileSystem) { /* appends unavailable: use create() paths only */ }

Try / catch

catch (IOException e) { if (e.getMessage().contains("Append is not supported")) { throw new UnsupportedOperationException("oss:// output must use create(), not append()", e); } throw e; }

Prevention

When it happens

Trigger: Calling FileSystem.append(path), FSDataOutputStream appends, or any engine operation that maps to append (e.g., Hive table appends writing to existing object files, HBase WAL-style appends, some StreamingFileSink configurations).

Common situations: Porting jobs written for HDFS that use append or lazy persist writes; Hive INSERT INTO tables whose writer attempts to append to existing files instead of creating new ones; frameworks defaulting to append semantics when the scheme is oss://.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/57531689949e38c8. Report an issue: GitHub.