apache/flink · error · IOException

Failed to put object for key: {}

Error message

Failed to put object for key: {}

What it means

putObject() is the single-request upload path used for smaller files (or when the TransferManager is disabled): it PUTs the local file with bucket, key and encryption headers. Any S3Exception from the service is wrapped in this IOException with the key. Unlike the multipart paths there is no uploadId involved; failure is atomic — either the object was written or nothing was.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3ObjectOperations.java:222

                    e);
        }
    }

    public PutObjectResult putObject(String key, File inputFile) throws IOException {
        if (useAsyncOperations && transferManager != null) {
            return putObjectViaTransferManager(key, inputFile);
        }

        try {
            PutObjectRequest.Builder requestBuilder =
                    PutObjectRequest.builder().bucket(bucketName).key(key);
            applyEncryption(requestBuilder);

            PutObjectResponse response =
                    s3Client.putObject(requestBuilder.build(), RequestBody.fromFile(inputFile));
            return new PutObjectResult(response.eTag());
        } catch (S3Exception e) {
            throw new IOException("Failed to put object for key: " + key, e);
        }
    }

    /**
     * Uploads an object using the S3TransferManager for better throughput.
     *
     * <p>The transfer manager provides optimizations over the basic S3 client:
     *
     * <ul>
     *   <li>Automatic multipart upload handling for large files
     *   <li>Optimized part sizes and parallelism
     *   <li>Better memory management through streaming
     * </ul>
     *
     * <p>This method blocks until the upload completes.
     */
    private PutObjectResult putObjectViaTransferManager(String key, File inputFile)
            throws IOException {

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Inspect the wrapped S3Exception code: 403 -> grant s3:PutObject (and kms:* if SSE-KMS); 404/301 -> fix bucket/endpoint/region/path-style.
2. Reproduce with the same credentials: aws s3api put-object --bucket B --key test --body file.
3. If files can exceed ~5GB, ensure the async/TransferManager path is enabled so it multiparts automatically (useAsyncOperations + transferManager).
4. For MinIO/compatible stores set fs.s3.endpoint + fs.s3.path.style.access: true + s3.region.

catch (IOException e) { S3Exception s3 = (S3Exception) e.getCause(); if (s3.statusCode() >= 500 || "SlowDown".equals(s3.awsErrorDetails().errorCode())) retryWithBackoff(); else surfaceIamOrConfigError(s3); }

When it happens

Trigger: IAM principal lacking s3:PutObject on the prefix; bucket missing or wrong endpoint; SSE-KMS key unusable by the role (kms:GenerateDataKey); object-lock/WORM bucket rejecting non-conforming headers; size exceeding single-PUT limits (~5GB) when the TransferManager path is disabled; expired credentials.

Common situations: Small in-progress part files or metadata objects written via putObject during commits; restrictive bucket policies added after jobs were running; encryption enabled between deploys without KMS grants; MinIO with path-style not set so the bucket is parsed as a hostname.

Related errors

• Failed to start multipart upload for key: {}
• fs.s3.aws.credentials.provider is set but contains no valid
• Class {} does not implement AwsCredentialsProvider
• Failed to instantiate credentials provider: {}
• AWS region could not be determined. Set 's3.region' in Flink

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/aa46a93e0ea460c2. Report an issue: GitHub.