apache/hadoop · error · UnsupportedOperationException

Multi-part uploader not supported for Client side encryption

Error message

Multi-part uploader not supported for Client side encryption.

What it means

createMultipartUploader() checks isCSEEnabled - set in initialize() when the configured encryption algorithm resolves to a client-side-encryption method (CSE-KMS/CSE-CUSTOM via fs.s3a.encryption.algorithm or the deprecated fs.s3a.server-side-encryption-algorithm) - and throws UnsupportedOperationException, because the MultipartUploader API cannot produce CSE-compatible uploads. Non-CSE filesystems get a working S3AMultipartUploaderBuilder; only CSE configurations are rejected.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:5590

    OpenFileSupport.OpenFileInformation fileInformation =
        openFileHelper.prepareToOpenFile(
            path,
            parameters,
            getDefaultBlockSize());
    CompletableFuture<FSDataInputStream> result = new CompletableFuture<>();
    unboundedThreadPool.submit(() ->
        LambdaUtils.eval(result,
            () -> executeOpen(path, fileInformation)));
    return result;
  }

  @Override
  @AuditEntryPoint
  public S3AMultipartUploaderBuilder createMultipartUploader(
      final Path basePath)
      throws IOException {
    if(isCSEEnabled) {
      throw new UnsupportedOperationException("Multi-part uploader not "
          + "supported for Client side encryption.");
    }
    final Path path = makeQualified(basePath);
    try (AuditSpan span = entryPoint(MULTIPART_UPLOAD_INSTANTIATED, path)) {
      StoreContext ctx = createStoreContext();
      return new S3AMultipartUploaderBuilder(this,
          createWriteOperationHelper(span),
          ctx,
          path,
          statisticsContext.createMultipartUploaderStatistics());
    }
  }

  /**
   * Build an immutable store context.
   * If called while the FS is being initialized,
   * some of the context will be incomplete.
   * new store context instances should be created as appropriate.

View on GitHub (pinned to 2add963021)

Solutions

Unset the CSE algorithm for workloads that need the MultipartUploader API (or move to SSE if policy allows)
Avoid the MultipartUploader API on CSE buckets - plain FSDataOutputStream writes through S3ABlockOutputStream still work
Separate buckets/configurations for CSE-only and multipart-uploader workloads

Example fix

<!-- before: CSE enabled, blocks multipart uploader -->
<property><name>fs.s3a.encryption.algorithm</name><value>CSE-KMS</value></property>
<property><name>fs.s3a.client.side.encryption.key</name><value>...</value></property>

<!-- after: SSE-KMS keeps server-side encryption compatible with multipart upload -->
<property><name>fs.s3a.encryption.algorithm</name><value>SSE-KMS</value></property>

Defensive patterns

Strategy: validation

Validate before calling

static boolean cseEnabled(Configuration conf) {
  String alg = conf.get("fs.s3a.encryption.algorithm",
      conf.get("fs.s3a.server-side-encryption-algorithm", ""));
  return alg.startsWith("CSE");
}

if (cseEnabled(conf)) {
  // skip the MultipartUploader API; use plain stream writes
}

Try / catch

Catch UnsupportedOperationException from createMultipartUploader() and fall back to normal FSDataOutputStream-based writes; surface the CSE/multipart incompatibility in configuration rather than disabling encryption silently.

Prevention

Keep CSE and multipart-uploader workloads on separate buckets/configurations
Detect CSE configuration before invoking MultipartUploader-based tooling
When encryption-at-rest and multipart APIs are both required, prefer SSE algorithms over CSE

When it happens

Trigger: Calling fs.createMultipartUploader(path), or running tools built on the MultipartUploader API (e.g. distcp multipart copy paths), on a filesystem whose encryption algorithm config selects CSE.

Common situations: CSE enabled for compliance on a bucket, then a tool or workflow that uses multipart upload APIs runs against the same configuration; core-site.xml with CSE settings shared across heterogeneous workloads.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a1c41f8e6f266107. Report an issue: GitHub.