What causes "Append is not supported by FTPFileSystem" in apache/hadoop?

Rewrite instead of append: read (or regenerate) the full content, concatenate locally, then fs.create(path, true) and write everything

apache/hadoop · error · UnsupportedOperationException

Append is not supported by FTPFileSystem

Error message

Append is not supported by FTPFileSystem

What it means

FTPFileSystem does not implement append(): per the FileSystem contract append is optional, and this implementation always throws UnsupportedOperationException. FTP's STOR-based write model used here has no server-side append.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java:381

        if (!client.isConnected()) {
          throw new FTPException("Client not connected");
        }
        boolean cmdCompleted = client.completePendingCommand();
        disconnect(client);
        if (!cmdCompleted) {
          throw new FTPException("Could not complete transfer, Reply Code - "
              + client.getReplyCode());
        }
      }
    };
    return fos;
  }

  /** This optional operation is not yet supported. */
  @Override
  public FSDataOutputStream append(Path f, int bufferSize,
      Progressable progress) throws IOException {
    throw new UnsupportedOperationException("Append is not supported "
        + "by FTPFileSystem");
  }
  
  /**
   * Convenience method, so that we don't open a new connection when using this
   * method from within another method. Otherwise every API invocation incurs
   * the overhead of opening/closing a TCP connection.
   * @throws IOException on IO problems other than FileNotFoundException
   */
  private boolean exists(FTPClient client, Path file) throws IOException {
    try {
      getFileStatus(client, file);
      return true;
    } catch (FileNotFoundException fnfe) {
      return false;
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

Rewrite instead of append: read (or regenerate) the full content, concatenate locally, then fs.create(path, true) and write everything
Use a filesystem that supports append (HDFS) for append-based workloads
Branch at runtime on the scheme (fs.getScheme().equals("ftp")) or catch UnsupportedOperationException to pick the rewrite strategy

Example fix

// before
FSDataOutputStream out = fs.append(path); // UnsupportedOperationException

// after - append by rewrite
byte[] existing = new byte[0];
if (fs.exists(path)) {
  try (FSDataInputStream in = fs.open(path)) {
    existing = IOUtils.toByteArray(in);
  }
}
try (FSDataOutputStream out = fs.create(path, true)) {
  out.write(existing);
  out.write(extraBytes);
}

Defensive patterns

Strategy: fallback

Validate before calling

boolean canAppend = !"ftp".equals(fs.getScheme());
if (!canAppend) {
  // use the rewrite path instead of fs.append()
}

Try / catch

try {
  out = fs.append(path);
} catch (UnsupportedOperationException e) {
  // fallback: read + rewrite
  out = rewriteWith(path, extraBytes, fs);
}

Prevention

Probe append support once per filesystem scheme and cache the decision
Design writers to create-or-rewrite rather than append when the scheme may be ftp
Prefer HDFS for workloads that genuinely require append semantics

When it happens

Trigger: Any fs.append(path) or fs.append(path, bufferSize) call on an ftp:// path — directly or through frameworks that rely on append (output-commit recovery, log tailing, append-based writers).

Common situations: Portable code that works on HDFS or s3a breaking when pointed at ftp://; frameworks probing append support at runtime; migration from HDFS to FTP-backed storage.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4f1b35edbfa3c870. Report an issue: GitHub.