apache/hadoop · error · UnsupportedOperationException

Append is not supported by SFTPFileSystem

Error message

Append is not supported by SFTPFileSystem

What it means

SFTPFileSystem.append unconditionally throws UnsupportedOperationException: the adapter implements no append, even though the SFTP wire protocol could express it via offset writes. Any code path that reaches append() on an sftp:// FileSystem fails.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java:595

    } catch (SftpException e) {
      throw new IOException(e);
    }
    FSDataOutputStream fos = new FSDataOutputStream(os, statistics) {
      @Override
      public void close() throws IOException {
        super.close();
        disconnect(client);
      }
    };

    return fos;
  }

  @Override
  public FSDataOutputStream append(Path f, int bufferSize,
      Progressable progress)
      throws IOException {
    throw new UnsupportedOperationException("Append is not supported "
        + "by SFTPFileSystem");
  }

  /*
   * The parent of source and destination can be different. It is suppose to
   * work like 'move'
   */
  @Override
  public boolean rename(Path src, Path dst) throws IOException {
    ChannelSftp channel = connect();
    try {
      boolean success = rename(channel, src, dst);
      return success;
    } finally {
      disconnect(channel);
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

  1. Replace append with read-modify-write: read the existing file, then create(f, true) and write old content followed by the new content.
  2. Write each batch to a new uniquely named file in the same directory and merge/consume them in order, avoiding in-place append entirely.
  3. Route paths that require true append semantics to a filesystem that supports it (e.g., HDFS) instead of sftp://.

Example fix

// before
FSDataOutputStream out = fs.append(logPath); // UnsupportedOperationException

// after
ByteArrayOutputStream buf = new ByteArrayOutputStream();
try (FSDataInputStream in = fs.open(logPath)) {
  IOUtils.copyBytes(in, buf, 4096, false);
}
try (FSDataOutputStream out = fs.create(logPath, true)) {
  buf.writeTo(out);
  out.write(newBytes);
}
Defensive patterns

Strategy: try-catch

Validate before calling

if (fs.getUri().getScheme().equals("sftp")) {
  // plan for read-modify-write; append() will throw UnsupportedOperationException
}

Try / catch

try {
  out = fs.append(f);
} catch (UnsupportedOperationException e) {
  // SFTP adapter: fall back to create(f, true) rewriting existing + new bytes
}

Prevention

When it happens

Trigger: Calling fs.append(f), fs.append(f, bufferSize), or fs.append(f, bufferSize, progress) on a FileSystem whose URI scheme is sftp; also framework code (writers, sinks, committers) that falls back to append for output finalization.

Common situations: Porting a pipeline from HDFS or the local filesystem where append works; log-aggregation style jobs that append to a single remote file; libraries that probe capabilities by calling append and catching failure.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c82fdc93b72e5ba0. Report an issue: GitHub.