What causes "Failed to async upload object for key: {}" in apache/flink?

Inspect the CAUSE chain (the original S3Exception) — this message alone does not carry the status code; the fix depends on the cause (IAM, throttling, timeout).

apache/flink · error · IOException

Failed to async upload object for key: {}

Error message

Failed to async upload object for key: {}

What it means

TransferManager variant of putObject: completionFuture().get() threw ExecutionException, meaning the async upload itself failed. The code unwraps e.getCause() (the actual SDK/runtime failure such as S3Exception, timeout, or credential error) into this IOException. The message distinguishes the async path from the sync putObject failure so you know the failure came through the S3TransferManager pipeline (its own retry/parallel machinery) rather than a direct API call.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3ObjectOperations.java:275

                                                            encryptionConfig
                                                                    .serializeEncryptionContext());
                                                }
                                            }
                                        }
                                    })
                            .source(inputFile.toPath())
                            .build();

            FileUpload fileUpload = transferManager.uploadFile(uploadRequest);
            CompletedFileUpload completedUpload;
            try {
                completedUpload = fileUpload.completionFuture().get();
            } catch (InterruptedException e) {
                fileUpload.completionFuture().cancel(true);
                Thread.currentThread().interrupt();
                throw new IOException("Interrupted while uploading object for key: " + key, e);
            } catch (ExecutionException e) {
                throw new IOException(
                        "Failed to async upload object for key: " + key, e.getCause());
            }
            return new PutObjectResult(completedUpload.response().eTag());
        } catch (IOException e) {
            throw e;
        } catch (Exception e) {
            throw new IOException("Failed to async upload object for key: " + key, e);
        }
    }

    /**
     * Completes a multipart upload by assembling previously uploaded parts.
     *
     * <p><b>Recovery Scenario:</b> If a {@link NoSuchUploadException} is thrown, this may indicate
     * that the upload was already completed (possibly by a previous attempt during recovery). In
     * this case, we check if the object exists and return its metadata. This handles the scenario
     * where:
     *

View on GitHub (pinned to 2f3c205e92)

Solutions

Inspect the CAUSE chain (the original S3Exception) — this message alone does not carry the status code; the fix depends on the cause (IAM, throttling, timeout).
For connection-acquisition timeouts: raise fs.s3.connection.maximum and/or lower sink parallelism; ensure async client's maxConcurrency matches expected concurrent transfers.
For throttling: enable/keep TransferManager (it retries), reduce parts-in-flight, or spread keys across prefixes.
For credentials: switch to a refreshing provider (instance profile) as static keys expired mid-upload.

Defensive patterns

Strategy: retry

Try / catch

catch (IOException e) { Throwable root = e.getCause(); if (root instanceof S3Exception && ((S3Exception) root).statusCode() >= 500) retryTransferWithBackoff(); else diagnoseCredentialsOrPermissions(root); }

Prevention

Match fs.s3.connection.maximum (async client maxConcurrency) to concurrent TransferManager transfers.
Use auto-refreshing credentials so long transfers never fail on expiry.
Log the unwrapped cause, not just this wrapper — the status code lives one level down.

When it happens

Trigger: Async upload failing after the TransferManager's internal retries are exhausted: S3 5xx/SlowDown, connection acquisition timeout from a too-small connection pool, expired credentials, KMS permission failures on encrypted uploads, local file read errors surfaced through the async pipeline.

Common situations: High writer parallelism saturating fs.s3.connection.maximum so Netty cannot hand out connections; mass recovery replaying many uploads; SSE-KMS grants missing; large part counts hitting per-request throttling.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/aabd1aa87a2a3a30. Report an issue: GitHub.