{"record":{"id":"1339b429be4768f8","repo":"apache/hadoop","slug":"append-is-not-supported-by-s3afilesystem","errorCode":null,"errorMessage":"Append is not supported by S3AFileSystem","messagePattern":"Append is not supported by S3AFileSystem","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java","lineNumber":2370,"sourceCode":"        .bufferSize(bufferSize)\n        .must(FS_S3A_CREATE_PERFORMANCE,\n            getPerformanceFlags().enabled(PerformanceFlagEnum.Create));\n    if (progress != null) {\n      builder.progress(progress);\n    }\n    return builder.build();\n  }\n\n  /**\n   * Append to an existing file (optional operation).\n   * @param f the existing file to be appended.\n   * @param bufferSize the size of the buffer to be used.\n   * @param progress for reporting progress if it is not null.\n   * @throws IOException indicating that append is not supported.\n   */\n  public FSDataOutputStream append(Path f, int bufferSize,\n      Progressable progress) throws IOException {\n    throw new UnsupportedOperationException(\"Append is not supported \"\n        + \"by S3AFileSystem\");\n  }\n\n\n  /**\n   * Renames Path src to Path dst.  Can take place on local fs\n   * or remote DFS.\n   *\n   * Warning: S3 does not support renames. This method does a copy which can\n   * take S3 some time to execute with large files and directories. Since\n   * there is no Progressable passed in, this can time out jobs.\n   *\n   * Note: This implementation differs with other S3 drivers. Specifically:\n   * <pre>\n   *       Fails if src is a file and dst is a directory.\n   *       Fails if src is a directory and dst is a file.\n   *       Fails if the parent of dst does not exist or is a file.\n   *       Fails if dst is a directory that is not empty.","sourceCodeStart":2352,"sourceCodeEnd":2388,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2352-L2388","documentation":"S3AFileSystem.append() unconditionally throws UnsupportedOperationException. S3 objects are immutable, so appending to an existing object cannot be implemented; rather than hide a slow full read-modify-write behind the append API, hadoop-aws rejects the optional FileSystem append contract outright.","triggerScenarios":"Any fs.append(path) or fs.append(path, bufferSize, progress) call on an s3a:// path; frameworks that probe append support by calling it.","commonSituations":"Log appenders, streaming sinks and HDFS-era job code ported to S3A; shared libraries that call append when a FileSystem does not advertise the capability.","solutions":["Restructure to write new objects per flush (suffixed or versioned names) and have readers pick the latest","Use HDFS or another mutable store for append-heavy workloads","If unavoidable, implement the read-modify-write yourself: copy the object plus new bytes to a new key, then swap - accepting the cost"],"exampleFix":"// before\ntry (FSDataOutputStream out = fs.append(logPath, 4096)) {\n  out.writeBytes(line);\n}\n\n// after: S3A has no append - write a new object\nPath next = new Path(logPath.getParent(),\n    logPath.getName() + \".\" + System.currentTimeMillis());\ntry (FSDataOutputStream out = fs.create(next, false)) {\n  out.writeBytes(line);\n}","handlingStrategy":"fallback","validationCode":"static boolean canAppend(FileSystem fs) {\n  String scheme = fs.getUri().getScheme();\n  return scheme == null || !(scheme.equals(\"s3a\") || scheme.equals(\"s3n\"));\n}","typeGuard":null,"tryCatchPattern":"Catch UnsupportedOperationException around append() and switch to the new-object write strategy; check fs.getUri().getScheme() first when the code must run on multiple filesystems.","preventionTips":["Never call append() on object-store schemes (s3a, abfs, gcs)","Design writers as create-new-object per flush","Run integration tests of IO helpers against the actual target filesystem"],"tags":["s3a","append","unsupported-operation","object-storage"],"backgroundTag":"append-not-supported","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}