apache/flink · error · IOException

Failed to start multipart upload for key

Error message

Failed to start multipart upload for key: {}

What it means

startMultiPartUpload() calls S3Client.createMultipartUpload (CreateMultipartUploadRequest with bucket, key, and server-side encryption applied per S3EncryptionConfig). Any S3Exception from the service — access denied, no such bucket, KMS key errors, invalid endpoint — is wrapped in this IOException with the object key. The uploadId this call returns is the handle for all subsequent part uploads, so nothing downstream can proceed until it succeeds.

Solutions

1. Read the nested S3Exception status/code: 403 AccessDenied -> fix IAM (s3:PutObject, s3:CreateMultipartUpload, and with SSE-KMS kms:GenerateDataKey+ kms:Decrypt on the key); 404 NoSuchBucket -> fix bucket name/endpoint.
2. Verify with aws s3api create-multipart-upload --bucket B --key test using the same credentials/role the Flink job uses.
3. If using SSE-KMS, confirm fs.s3.kms-key.id points to a key the role can use and that the key is enabled in the region.
4. Check endpoint/region/path-style settings match the bucket (MinIO: fs.s3.path.style.access: true + endpoint + s3.region).

Example fix

# before: policy grants only s3:PutObject
{ "Effect": "Allow", "Action": ["s3:PutObject"], "Resource": "arn:aws:s3:::my-bucket/*" }

# after: include multipart actions
{ "Effect": "Allow",
  "Action": ["s3:PutObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts", "s3:ListBucketMultipartUploads"],
  "Resource": ["arn:aws:s3:::my-bucket/*", "arn:aws:s3:::my-bucket"] }

// Pre-flight permission probe with the job's credentials
try (S3Client c = S3Client.create()) {
    c.createMultipartUpload(b -> b.bucket(bucket).key("__flink_probe__"));
    // abort immediately; failure here reproduces the IAM problem before a job runs
}

catch (IOException e) { S3Exception s3 = (S3Exception) e.getCause(); switch (s3.statusCode()) { case 403: fail IAM audit; case 404: fix bucket/endpoint; case 500/503: retry with backoff; default: rethrow; } }

When it happens

Trigger: IAM role missing s3:PutObject / s3:CreateMultipartUpload on the bucket or prefix; bucket in another account without cross-account policy; SSE-KMS configured but the role lacks kms:GenerateDataKey on the CMK (CreateMultipartUpload is where SSE headers are validated); wrong bucket name or endpoint override; bucket-owner-enforced-object-writes with wrong principal.

Common situations: First write to a new bucket with restrictive policy; enabling fs.s3.encryption without granting the Flink role KMS permissions; typos in fs.s3.bucket or endpoint; VPC endpoint policies blocking multipart creation; expired temporary credentials.

Related errors

• Failed to abort multipart upload for key
• Failed to put object for key
• Failed to upload part
• Cannot commit empty multipart upload for object
• Failed to async upload object for key

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/5b03487f3d2c742f. Report an issue: GitHub.

Appendix: source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3ObjectOperations.java:147

        this.s3Client = s3Client;
        this.transferManager = transferManager;
        this.bucketName = bucketName;
        this.useAsyncOperations = useAsyncOperations && transferManager != null;
        this.encryptionConfig =
                encryptionConfig != null ? encryptionConfig : S3EncryptionConfig.none();
    }

    public String startMultiPartUpload(String key) throws IOException {
        try {
            CreateMultipartUploadRequest.Builder requestBuilder =
                    CreateMultipartUploadRequest.builder().bucket(bucketName).key(key);
            applyEncryption(requestBuilder);

            CreateMultipartUploadResponse response =
                    s3Client.createMultipartUpload(requestBuilder.build());
            return response.uploadId();
        } catch (S3Exception e) {
            throw new IOException("Failed to start multipart upload for key: " + key, e);
        }
    }

    private void applyEncryption(CreateMultipartUploadRequest.Builder requestBuilder) {
        if (!encryptionConfig.isEnabled()) {
            return;
        }
        requestBuilder.serverSideEncryption(encryptionConfig.getServerSideEncryption());
        if (encryptionConfig.getEncryptionType() == S3EncryptionConfig.EncryptionType.SSE_KMS) {
            if (encryptionConfig.getKmsKeyId() != null) {
                requestBuilder.ssekmsKeyId(encryptionConfig.getKmsKeyId());
            }
            if (encryptionConfig.hasEncryptionContext()) {
                requestBuilder.ssekmsEncryptionContext(
                        encryptionConfig.serializeEncryptionContext());
            }
        }
    }

View on GitHub (pinned to 2f3c205e92)