{"record":{"id":"46855d00d81c7684","repo":"apache/hadoop","slug":"unable-to-create-outputstream-with-the-given-multi","errorCode":null,"errorMessage":"Unable to create OutputStream with the given multipart upload and buffer configuration.","messagePattern":"Unable to create OutputStream with the given multipart upload and buffer configuration\\.","errorType":"exception","errorClass":"PathIOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java","lineNumber":1018,"sourceCode":"    long partSize = conf.getLongBytes(property, defVal);\n    if (partSize < MULTIPART_MIN_SIZE) {\n      LOG.warn(\"{} must be at least 5 MB; configured value is {}\",\n          property, partSize);\n      partSize = MULTIPART_MIN_SIZE;\n    }\n    return partSize;\n  }\n\n  /**\n   * Validates the output stream configuration.\n   * @param path path: for error messages\n   * @param conf : configuration object for the given context\n   * @throws PathIOException Unsupported configuration.\n   */\n  public static void validateOutputStreamConfiguration(final Path path,\n      Configuration conf) throws PathIOException {\n    if(!checkDiskBuffer(conf)){\n      throw new PathIOException(path.toString(),\n          \"Unable to create OutputStream with the given\"\n          + \" multipart upload and buffer configuration.\");\n    }\n  }\n\n  /**\n   * Check whether the configuration for S3ABlockOutputStream is\n   * consistent or not. Multipart uploads allow all kinds of fast buffers to\n   * be supported. When the option is disabled only disk buffers are allowed to\n   * be used as the file size might be bigger than the buffer size that can be\n   * allocated.\n   * @param conf : configuration object for the given context\n   * @return true if the disk buffer and the multipart settings are supported\n   */\n  public static boolean checkDiskBuffer(Configuration conf) {\n    boolean isMultipartUploadEnabled = conf.getBoolean(MULTIPART_UPLOADS_ENABLED,\n        DEFAULT_MULTIPART_UPLOAD_ENABLED);\n    return isMultipartUploadEnabled","sourceCodeStart":1000,"sourceCodeEnd":1036,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L1000-L1036","documentation":"PathIOException from S3AUtils.validateOutputStreamConfiguration, raised when checkDiskBuffer() returns false: multipart uploads are disabled (fs.s3a.multipart.uploads.enabled=false) AND fs.s3a.fast.upload.buffer is not 'disk'. Without multipart upload the whole file must be buffered locally until a single putObject, so only the disk buffer scales beyond heap; multipart-enabled setups may use any fast buffer.","triggerScenarios":"Creating an S3A output stream (fs.create()) with fs.s3a.multipart.uploads.enabled=false while fs.s3a.fast.upload.buffer is explicitly set to array or bytebuffer. The check runs when the output stream is built.","commonSituations":"Sites that disabled multipart uploads (to block pending-upload debris or committers) without switching the buffer to disk; inherited fast-upload defaults meeting a globally disabled multipart flag; per-bucket overrides diverging from the global config.","solutions":["Re-enable multipart uploads: set fs.s3a.multipart.uploads.enabled=true (the default, and required by the S3A committers and fast upload buffers)","Or keep multipart disabled and set fs.s3a.fast.upload.buffer=disk so writes spill to local disk","Verify with a small write test after the config change","Document the coupling: non-disk fast-upload buffers are only valid with multipart uploads enabled"],"exampleFix":"<!-- before: multipart off + non-disk buffer -->\n<property><name>fs.s3a.multipart.uploads.enabled</name><value>false</value></property>\n<property><name>fs.s3a.fast.upload.buffer</name><value>bytebuffer</value></property>\n\n<!-- after: pick one coherent policy -->\n<!-- A: restore multipart (recommended) -->\n<property><name>fs.s3a.multipart.uploads.enabled</name><value>true</value></property>\n<!-- B: keep multipart off, buffer to disk instead:\n<property><name>fs.s3a.fast.upload.buffer</name><value>disk</value></property> -->","handlingStrategy":"validation","validationCode":"boolean multipart = conf.getBoolean(\"fs.s3a.multipart.uploads.enabled\", true);\nString buffer = conf.get(\"fs.s3a.fast.upload.buffer\", \"disk\");\nif (!multipart && !\"disk\".equals(buffer)) {\n  throw new IOException(\"Invalid S3A output config: with multipart uploads disabled,\"\n      + \" fs.s3a.fast.upload.buffer must be 'disk' (is: \" + buffer + \")\");\n}","typeGuard":null,"tryCatchPattern":"catch PathIOException at fs.create() containing 'Unable to create OutputStream'; fail fast with config guidance - the combination is invalid, retrying cannot help","preventionTips":["Treat fs.s3a.multipart.uploads.enabled=false as an expert-only switch always paired with fs.s3a.fast.upload.buffer=disk","Add a config lint step validating this pair before deployment","Remember the S3A committers also require multipart enabled"],"tags":["s3a","hadoop-aws","configuration","output-stream","multipart-upload","pathioexception"],"backgroundTag":"invalid-configuration","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}