{"record":{"id":"c82fdc93b72e5ba0","repo":"apache/hadoop","slug":"append-is-not-supported-by-sftpfilesystem","errorCode":null,"errorMessage":"Append is not supported by SFTPFileSystem","messagePattern":"Append is not supported by SFTPFileSystem","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java","lineNumber":595,"sourceCode":"    } catch (SftpException e) {\n      throw new IOException(e);\n    }\n    FSDataOutputStream fos = new FSDataOutputStream(os, statistics) {\n      @Override\n      public void close() throws IOException {\n        super.close();\n        disconnect(client);\n      }\n    };\n\n    return fos;\n  }\n\n  @Override\n  public FSDataOutputStream append(Path f, int bufferSize,\n      Progressable progress)\n      throws IOException {\n    throw new UnsupportedOperationException(\"Append is not supported \"\n        + \"by SFTPFileSystem\");\n  }\n\n  /*\n   * The parent of source and destination can be different. It is suppose to\n   * work like 'move'\n   */\n  @Override\n  public boolean rename(Path src, Path dst) throws IOException {\n    ChannelSftp channel = connect();\n    try {\n      boolean success = rename(channel, src, dst);\n      return success;\n    } finally {\n      disconnect(channel);\n    }\n  }\n","sourceCodeStart":577,"sourceCodeEnd":613,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java#L577-L613","documentation":"SFTPFileSystem.append unconditionally throws UnsupportedOperationException: the adapter implements no append, even though the SFTP wire protocol could express it via offset writes. Any code path that reaches append() on an sftp:// FileSystem fails.","triggerScenarios":"Calling fs.append(f), fs.append(f, bufferSize), or fs.append(f, bufferSize, progress) on a FileSystem whose URI scheme is sftp; also framework code (writers, sinks, committers) that falls back to append for output finalization.","commonSituations":"Porting a pipeline from HDFS or the local filesystem where append works; log-aggregation style jobs that append to a single remote file; libraries that probe capabilities by calling append and catching failure.","solutions":["Replace append with read-modify-write: read the existing file, then create(f, true) and write old content followed by the new content.","Write each batch to a new uniquely named file in the same directory and merge/consume them in order, avoiding in-place append entirely.","Route paths that require true append semantics to a filesystem that supports it (e.g., HDFS) instead of sftp://."],"exampleFix":"// before\nFSDataOutputStream out = fs.append(logPath); // UnsupportedOperationException\n\n// after\nByteArrayOutputStream buf = new ByteArrayOutputStream();\ntry (FSDataInputStream in = fs.open(logPath)) {\n  IOUtils.copyBytes(in, buf, 4096, false);\n}\ntry (FSDataOutputStream out = fs.create(logPath, true)) {\n  buf.writeTo(out);\n  out.write(newBytes);\n}","handlingStrategy":"try-catch","validationCode":"if (fs.getUri().getScheme().equals(\"sftp\")) {\n  // plan for read-modify-write; append() will throw UnsupportedOperationException\n}","typeGuard":null,"tryCatchPattern":"try {\n  out = fs.append(f);\n} catch (UnsupportedOperationException e) {\n  // SFTP adapter: fall back to create(f, true) rewriting existing + new bytes\n}","preventionTips":["Do not design pipelines that require in-place append over sftp://.","Write to new uniquely named files and merge downstream.","Catch UnsupportedOperationException (unchecked) when generic code must run across filesystems."],"tags":["sftp","append","unsupported-operation","hadoop","filesystem"],"backgroundTag":"unsupported-filesystem-operation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}