apache/flink · error · IOException

Failed to upload part %d for key: %s, uploadId: %s

Error message

Failed to upload part %d for key: %s, uploadId: %s

What it means

uploadPart() sends one already-materialized part file (RequestBody.fromFile) as UploadPart with a given partNumber and uploadId, returning the part's eTag for later CompleteMultipartUpload. Any S3Exception during the call is wrapped into this formatted IOException including part number, key and uploadId — the three identifiers needed to either retry the single part or abort the upload.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3ObjectOperations.java:200

    public UploadPartResult uploadPart(
            String key, String uploadId, int partNumber, File inputFile, long length)
            throws IOException {
        try {
            UploadPartRequest request =
                    UploadPartRequest.builder()
                            .bucket(bucketName)
                            .key(key)
                            .uploadId(uploadId)
                            .partNumber(partNumber)
                            .build();

            UploadPartResponse response =
                    s3Client.uploadPart(request, RequestBody.fromFile(inputFile));

            return new UploadPartResult(partNumber, response.eTag());
        } catch (S3Exception e) {
            throw new IOException(
                    String.format(
                            "Failed to upload part %d for key: %s, uploadId: %s",
                            partNumber, key, uploadId),
                    e);
        }
    }

    public PutObjectResult putObject(String key, File inputFile) throws IOException {
        if (useAsyncOperations && transferManager != null) {
            return putObjectViaTransferManager(key, inputFile);
        }

        try {
            PutObjectRequest.Builder requestBuilder =
                    PutObjectRequest.builder().bucket(bucketName).key(key);
            applyEncryption(requestBuilder);

            PutObjectResponse response =

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Retry the whole recoverable upload: Flink's writer semantics allow re-uploading the same part number with a new etag — or better, recover via the RecoverableWriter which re-uploads from the persisted offset.
2. If cause is NoSuchUpload, restart from a fresh startMultiPartUpload (the old upload was completed/aborted by another attempt).
3. For SlowDown/503: reduce fs.s3.limit.outside/inside? — concretely lower writer parallelism or part-upload concurrency and ensure fs.s3.connection.maximum is sized for parallel parts.
4. For expired credentials: use a credentials provider that refreshes (instance profile / DynamicTemporaryAWSCredentialsProvider) instead of static keys.

Example fix

// before — one-shot part upload with no retry
UploadPartResult r = ops.uploadPart(key, uploadId, partNumber, file);

// after — retry the individual part (same partNumber is safe; a new etag supersedes)
UploadPartResult r = retryOnS3Transient(
    () -> ops.uploadPart(key, uploadId, partNumber, file),
    5, Duration.ofSeconds(2));

catch (IOException e) {
    Throwable c = e.getCause();
    if (c instanceof NoSuchUploadException) { restartUploadFromScratch(); }
    else if (c instanceof S3Exception && ((S3Exception) c).statusCode() >= 500) { retrySamePartWithBackoff(); } // same partNumber supersedes prior etag
    else throw e;
}

When it happens

Trigger: Network interruption or 5xx/SlowDown during upload of a large part (default part sizes are multi-MB); credentials expiring mid-upload (session/STS); uploadId already aborted/completed (NoSuchUpload surfaces here if a sibling attempt finished it); throttling on high-parallelism part uploads; part file deleted locally before the call reads it (non-S3 error would differ, but fromFile read failure can surface inside).

Common situations: Long checkpoint uploads where STS session expires halfway; too many concurrent streams overwhelming connection pool causing timeouts; transient AWS-side throttling during mass recovery; recovery races where another attempt aborted the upload.

Related errors

• Failed to start multipart upload for key: {}
• Failed to async upload object for key: {}
• fs.s3.aws.credentials.provider is set but contains no valid
• Class {} does not implement AwsCredentialsProvider
• Failed to instantiate credentials provider: {}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/31004f29ad2b606d. Report an issue: GitHub.