{"record":{"id":"aabd1aa87a2a3a30","repo":"apache/flink","slug":"failed-to-async-upload-object-for-key","errorCode":null,"errorMessage":"Failed to async upload object for key: {}","messagePattern":"Failed to async upload object for key: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3ObjectOperations.java","lineNumber":275,"sourceCode":"                                                            encryptionConfig\n                                                                    .serializeEncryptionContext());\n                                                }\n                                            }\n                                        }\n                                    })\n                            .source(inputFile.toPath())\n                            .build();\n\n            FileUpload fileUpload = transferManager.uploadFile(uploadRequest);\n            CompletedFileUpload completedUpload;\n            try {\n                completedUpload = fileUpload.completionFuture().get();\n            } catch (InterruptedException e) {\n                fileUpload.completionFuture().cancel(true);\n                Thread.currentThread().interrupt();\n                throw new IOException(\"Interrupted while uploading object for key: \" + key, e);\n            } catch (ExecutionException e) {\n                throw new IOException(\n                        \"Failed to async upload object for key: \" + key, e.getCause());\n            }\n            return new PutObjectResult(completedUpload.response().eTag());\n        } catch (IOException e) {\n            throw e;\n        } catch (Exception e) {\n            throw new IOException(\"Failed to async upload object for key: \" + key, e);\n        }\n    }\n\n    /**\n     * Completes a multipart upload by assembling previously uploaded parts.\n     *\n     * <p><b>Recovery Scenario:</b> If a {@link NoSuchUploadException} is thrown, this may indicate\n     * that the upload was already completed (possibly by a previous attempt during recovery). In\n     * this case, we check if the object exists and return its metadata. This handles the scenario\n     * where:\n     *","sourceCodeStart":257,"sourceCodeEnd":293,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3ObjectOperations.java#L257-L293","documentation":"TransferManager variant of putObject: completionFuture().get() threw ExecutionException, meaning the async upload itself failed. The code unwraps e.getCause() (the actual SDK/runtime failure such as S3Exception, timeout, or credential error) into this IOException. The message distinguishes the async path from the sync putObject failure so you know the failure came through the S3TransferManager pipeline (its own retry/parallel machinery) rather than a direct API call.","triggerScenarios":"Async upload failing after the TransferManager's internal retries are exhausted: S3 5xx/SlowDown, connection acquisition timeout from a too-small connection pool, expired credentials, KMS permission failures on encrypted uploads, local file read errors surfaced through the async pipeline.","commonSituations":"High writer parallelism saturating fs.s3.connection.maximum so Netty cannot hand out connections; mass recovery replaying many uploads; SSE-KMS grants missing; large part counts hitting per-request throttling.","solutions":["Inspect the CAUSE chain (the original S3Exception) — this message alone does not carry the status code; the fix depends on the cause (IAM, throttling, timeout).","For connection-acquisition timeouts: raise fs.s3.connection.maximum and/or lower sink parallelism; ensure async client's maxConcurrency matches expected concurrent transfers.","For throttling: enable/keep TransferManager (it retries), reduce parts-in-flight, or spread keys across prefixes.","For credentials: switch to a refreshing provider (instance profile) as static keys expired mid-upload."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (IOException e) { Throwable root = e.getCause(); if (root instanceof S3Exception && ((S3Exception) root).statusCode() >= 500) retryTransferWithBackoff(); else diagnoseCredentialsOrPermissions(root); }","preventionTips":["Match fs.s3.connection.maximum (async client maxConcurrency) to concurrent TransferManager transfers.","Use auto-refreshing credentials so long transfers never fail on expiry.","Log the unwrapped cause, not just this wrapper — the status code lives one level down."],"tags":["s3","transfer-manager","async","network","retry","flink"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}