apache/hadoop · error · PathIOException

Unable to create OutputStream with the given multipart uploa

Error message

Unable to create OutputStream with the given multipart upload and buffer configuration.

What it means

PathIOException from S3AUtils.validateOutputStreamConfiguration, raised when checkDiskBuffer() returns false: multipart uploads are disabled (fs.s3a.multipart.uploads.enabled=false) AND fs.s3a.fast.upload.buffer is not 'disk'. Without multipart upload the whole file must be buffered locally until a single putObject, so only the disk buffer scales beyond heap; multipart-enabled setups may use any fast buffer.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java:1018

    long partSize = conf.getLongBytes(property, defVal);
    if (partSize < MULTIPART_MIN_SIZE) {
      LOG.warn("{} must be at least 5 MB; configured value is {}",
          property, partSize);
      partSize = MULTIPART_MIN_SIZE;
    }
    return partSize;
  }

  /**
   * Validates the output stream configuration.
   * @param path path: for error messages
   * @param conf : configuration object for the given context
   * @throws PathIOException Unsupported configuration.
   */
  public static void validateOutputStreamConfiguration(final Path path,
      Configuration conf) throws PathIOException {
    if(!checkDiskBuffer(conf)){
      throw new PathIOException(path.toString(),
          "Unable to create OutputStream with the given"
          + " multipart upload and buffer configuration.");
    }
  }

  /**
   * Check whether the configuration for S3ABlockOutputStream is
   * consistent or not. Multipart uploads allow all kinds of fast buffers to
   * be supported. When the option is disabled only disk buffers are allowed to
   * be used as the file size might be bigger than the buffer size that can be
   * allocated.
   * @param conf : configuration object for the given context
   * @return true if the disk buffer and the multipart settings are supported
   */
  public static boolean checkDiskBuffer(Configuration conf) {
    boolean isMultipartUploadEnabled = conf.getBoolean(MULTIPART_UPLOADS_ENABLED,
        DEFAULT_MULTIPART_UPLOAD_ENABLED);
    return isMultipartUploadEnabled

View on GitHub (pinned to 2add963021)

Solutions

1. Re-enable multipart uploads: set fs.s3a.multipart.uploads.enabled=true (the default, and required by the S3A committers and fast upload buffers)
2. Or keep multipart disabled and set fs.s3a.fast.upload.buffer=disk so writes spill to local disk
3. Verify with a small write test after the config change
4. Document the coupling: non-disk fast-upload buffers are only valid with multipart uploads enabled

Example fix

<!-- before: multipart off + non-disk buffer -->
<property><name>fs.s3a.multipart.uploads.enabled</name><value>false</value></property>
<property><name>fs.s3a.fast.upload.buffer</name><value>bytebuffer</value></property>

<!-- after: pick one coherent policy -->
<!-- A: restore multipart (recommended) -->
<property><name>fs.s3a.multipart.uploads.enabled</name><value>true</value></property>
<!-- B: keep multipart off, buffer to disk instead:
<property><name>fs.s3a.fast.upload.buffer</name><value>disk</value></property> -->

boolean multipart = conf.getBoolean("fs.s3a.multipart.uploads.enabled", true);
String buffer = conf.get("fs.s3a.fast.upload.buffer", "disk");
if (!multipart && !"disk".equals(buffer)) {
  throw new IOException("Invalid S3A output config: with multipart uploads disabled,"
      + " fs.s3a.fast.upload.buffer must be 'disk' (is: " + buffer + ")");
}

catch PathIOException at fs.create() containing 'Unable to create OutputStream'; fail fast with config guidance - the combination is invalid, retrying cannot help

When it happens

Trigger: Creating an S3A output stream (fs.create()) with fs.s3a.multipart.uploads.enabled=false while fs.s3a.fast.upload.buffer is explicitly set to array or bytebuffer. The check runs when the output stream is built.

Common situations: Sites that disabled multipart uploads (to block pending-upload debris or committers) without switching the buffer to disk; inherited fast-upload defaults meeting a globally disabled multipart flag; per-bucket overrides diverging from the global config.

Related errors

• No upload parts in multipart upload
• Multipart IO request {sdkRequest} rejected {header}
• Cannot find password option {key}
• SSE-C is enabled but no encryption key was declared in fs.s3
• AES256 is enabled but an encryption key was set in fs.s3a.en

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/46855d00d81c7684. Report an issue: GitHub.