{"record":{"id":"133b0bafdce5d25e","repo":"apache/flink","slug":"failed-to-get-object-for-key","errorCode":null,"errorMessage":"Failed to get object for key: {}","messagePattern":"Failed to get object for key: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3ObjectOperations.java","lineNumber":418,"sourceCode":"        java.nio.file.Path target = targetLocation.toPath().toAbsolutePath();\n        java.nio.file.Path parent = target.getParent();\n        if (parent != null) {\n            Files.createDirectories(parent);\n        }\n        java.nio.file.Path tempTarget =\n                NativeS3FileIoUtils.createTemporaryDownloadFile(parent, target);\n        ResponseInputStream<GetObjectResponse> responseStream = null;\n        boolean success = false;\n        try {\n            GetObjectRequest request =\n                    GetObjectRequest.builder().bucket(bucketName).key(key).build();\n            responseStream = s3Client.getObject(request);\n            NativeS3FileIoUtils.copyStream(responseStream, tempTarget, DOWNLOAD_BUFFER_SIZE);\n            NativeS3FileIoUtils.moveFile(tempTarget, target);\n            success = true;\n            return Files.size(target);\n        } catch (S3Exception e) {\n            throw new IOException(\"Failed to get object for key: \" + key, e);\n        } finally {\n            if (success) {\n                IOUtils.closeQuietly(responseStream);\n            } else {\n                abortAndClose(responseStream);\n                IOUtils.deleteFileQuietly(tempTarget);\n            }\n        }\n    }\n\n    private static void abortAndClose(ResponseInputStream<GetObjectResponse> stream) {\n        if (stream == null) {\n            return;\n        }\n        try {\n            stream.abort();\n        } catch (RuntimeException e) {\n            LOG.debug(\"Error aborting S3 response stream\", e);","sourceCodeStart":400,"sourceCodeEnd":436,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3ObjectOperations.java#L400-L436","documentation":"Thrown by NativeS3ObjectOperations.getObject when an AWS SDK S3Exception escapes while downloading an object via GetObject. The S3 GET either failed at the HTTP layer or was rejected by S3, and the original exception is wrapped in an IOException with the requested key. On failure the code aborts the response stream and deletes the temporary download file, so no partial target file remains.","triggerScenarios":"Calling s3AccessHelper.getObject(key, target) (directly or via NativeS3RecoverableWriter.downloadIncompleteTail during recover()) when: the key does not exist (404 NoSuchKey), credentials are missing/expired, the bucket is unreachable, a network/timeout error occurs mid-transfer, or S3 returns 403/5xx. Any S3Exception from s3Client.getObject(request) or the subsequent stream copy triggers it.","commonSituations":"Recovering a S3 recoverable stream from a checkpoint whose incomplete-tail side object was deleted by lifecycle rules or manually; wrong s3.access-key/secret in flink-conf; IMDS credentials expired on EC2; S3 bucket in a different region endpoint; transient network blips in long-running jobs.","solutions":["Check the nested S3Exception (getCause()) for the AWS error code (NoSuchKey, AccessDenied, RequestTimeout) and address that first.","Verify the object key still exists: aws s3api head-object --bucket <bucket> --key <key>.","If credentials/permission related, fix s3.access-key / s3.secret-key / bucket policies in the Flink S3 config.","For transient network or 5xx errors, retry the operation; the temp download file is cleaned up automatically so retry is safe.","If the side object was permanently deleted (lifecycle rule), the checkpoint is unusable for resume — restore from an earlier checkpoint that does not reference it."],"exampleFix":"// before\nlong len = s3AccessHelper.getObject(key, targetFile);\n\n// after\nlong len;\ntry {\n    len = s3AccessHelper.getObject(key, targetFile);\n} catch (IOException e) {\n    if (e.getCause() instanceof S3Exception\n            && ((S3Exception) e.getCause()).statusCode() == 404) {\n        throw new IOException(\"Side object vanished: \" + key\n                + \" — checkpoint cannot be resumed\", e);\n    }\n    throw e; // transient — let Flink retry the recovery","handlingStrategy":"retry","validationCode":"try (ResponseInputStream<GetObjectResponse> probe = s3Client.getObject(\n        GetObjectRequest.builder().bucket(bucket).key(key).build())) {\n    // object is readable — proceed with the real download\n}","typeGuard":null,"tryCatchPattern":"try {\n    long len = s3AccessHelper.getObject(key, target);\n} catch (IOException e) {\n    S3Exception cause = (S3Exception) e.getCause();\n    if (cause != null && cause.statusCode() >= 500 || cause.isRetryable()) {\n        // transient: allow Flink/task retry\n        throw e;\n    }\n    if (cause != null && cause.statusCode() == 404) {\n        // permanent: checkpoint unusable, roll back\n        throw new IOException(\"Side object gone, restore earlier checkpoint\", e);\n    }\n    throw e;\n}","preventionTips":["Exclude recoverable-state prefixes from S3 lifecycle expiration rules.","Monitor for 404/403 S3 errors in task logs as an early signal.","Keep S3 credentials configured via Flink's s3.* options, not ambient environment."],"tags":["s3","filesystem","network","io","recovery"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}