apache/flink · error · IOException

Failed to get object for key: {}

Error message

Failed to get object for key: {}

What it means

Thrown by NativeS3ObjectOperations.getObject when an AWS SDK S3Exception escapes while downloading an object via GetObject. The S3 GET either failed at the HTTP layer or was rejected by S3, and the original exception is wrapped in an IOException with the requested key. On failure the code aborts the response stream and deletes the temporary download file, so no partial target file remains.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3ObjectOperations.java:418

        java.nio.file.Path target = targetLocation.toPath().toAbsolutePath();
        java.nio.file.Path parent = target.getParent();
        if (parent != null) {
            Files.createDirectories(parent);
        }
        java.nio.file.Path tempTarget =
                NativeS3FileIoUtils.createTemporaryDownloadFile(parent, target);
        ResponseInputStream<GetObjectResponse> responseStream = null;
        boolean success = false;
        try {
            GetObjectRequest request =
                    GetObjectRequest.builder().bucket(bucketName).key(key).build();
            responseStream = s3Client.getObject(request);
            NativeS3FileIoUtils.copyStream(responseStream, tempTarget, DOWNLOAD_BUFFER_SIZE);
            NativeS3FileIoUtils.moveFile(tempTarget, target);
            success = true;
            return Files.size(target);
        } catch (S3Exception e) {
            throw new IOException("Failed to get object for key: " + key, e);
        } finally {
            if (success) {
                IOUtils.closeQuietly(responseStream);
            } else {
                abortAndClose(responseStream);
                IOUtils.deleteFileQuietly(tempTarget);
            }
        }
    }

    private static void abortAndClose(ResponseInputStream<GetObjectResponse> stream) {
        if (stream == null) {
            return;
        }
        try {
            stream.abort();
        } catch (RuntimeException e) {
            LOG.debug("Error aborting S3 response stream", e);

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Check the nested S3Exception (getCause()) for the AWS error code (NoSuchKey, AccessDenied, RequestTimeout) and address that first.
2. Verify the object key still exists: aws s3api head-object --bucket <bucket> --key <key>.
3. If credentials/permission related, fix s3.access-key / s3.secret-key / bucket policies in the Flink S3 config.
4. For transient network or 5xx errors, retry the operation; the temp download file is cleaned up automatically so retry is safe.
5. If the side object was permanently deleted (lifecycle rule), the checkpoint is unusable for resume — restore from an earlier checkpoint that does not reference it.

Example fix

// before
long len = s3AccessHelper.getObject(key, targetFile);

// after
long len;
try {
    len = s3AccessHelper.getObject(key, targetFile);
} catch (IOException e) {
    if (e.getCause() instanceof S3Exception
            && ((S3Exception) e.getCause()).statusCode() == 404) {
        throw new IOException("Side object vanished: " + key
                + " — checkpoint cannot be resumed", e);
    }
    throw e; // transient — let Flink retry the recovery

try (ResponseInputStream<GetObjectResponse> probe = s3Client.getObject(
        GetObjectRequest.builder().bucket(bucket).key(key).build())) {
    // object is readable — proceed with the real download
}

try {
    long len = s3AccessHelper.getObject(key, target);
} catch (IOException e) {
    S3Exception cause = (S3Exception) e.getCause();
    if (cause != null && cause.statusCode() >= 500 || cause.isRetryable()) {
        // transient: allow Flink/task retry
        throw e;
    }
    if (cause != null && cause.statusCode() == 404) {
        // permanent: checkpoint unusable, roll back
        throw new IOException("Side object gone, restore earlier checkpoint", e);
    }
    throw e;
}

When it happens

Trigger: Calling s3AccessHelper.getObject(key, target) (directly or via NativeS3RecoverableWriter.downloadIncompleteTail during recover()) when: the key does not exist (404 NoSuchKey), credentials are missing/expired, the bucket is unreachable, a network/timeout error occurs mid-transfer, or S3 returns 403/5xx. Any S3Exception from s3Client.getObject(request) or the subsequent stream copy triggers it.

Common situations: Recovering a S3 recoverable stream from a checkpoint whose incomplete-tail side object was deleted by lifecycle rules or manually; wrong s3.access-key/secret in flink-conf; IMDS credentials expired on EC2; S3 bucket in a different region endpoint; transient network blips in long-running jobs.

Related errors

• Incomplete-tail file does not exist: {}
• Failed to get metadata for key: {}
• Native S3 File System cannot recover recoverable for other f
• An I/O error occurred while creating temporary file to extra
• Failed to create parent(s) for given base dir: %s

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/133b0bafdce5d25e. Report an issue: GitHub.