{"record":{"id":"40b2878c0483480e","repo":"apache/flink","slug":"error-recovering-writer-downloading-the-last-data-40b287","errorCode":null,"errorMessage":"Error recovering writer: Downloading the last data chunk file gives incorrect length. File=%d bytes, Stream=%d bytes","messagePattern":"Error recovering writer: Downloading the last data chunk file gives incorrect length\\. File=(.+?) bytes, Stream=(.+?) bytes","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/HadoopS3AccessHelper.java","lineNumber":130,"sourceCode":"\n    @Override\n    public long getObject(String key, File targetLocation) throws IOException {\n        long numBytes = 0L;\n        try (final OutputStream outStream = new FileOutputStream(targetLocation);\n                final org.apache.hadoop.fs.FSDataInputStream inStream =\n                        s3a.open(new org.apache.hadoop.fs.Path('/' + key))) {\n            final byte[] buffer = new byte[32 * 1024];\n\n            int numRead;\n            while ((numRead = inStream.read(buffer)) != -1) {\n                outStream.write(buffer, 0, numRead);\n                numBytes += numRead;\n            }\n        }\n\n        // some sanity checks\n        if (numBytes != targetLocation.length()) {\n            throw new IOException(\n                    String.format(\n                            \"Error recovering writer: \"\n                                    + \"Downloading the last data chunk file gives incorrect length. \"\n                                    + \"File=%d bytes, Stream=%d bytes\",\n                            targetLocation.length(), numBytes));\n        }\n\n        return numBytes;\n    }\n\n    @Override\n    public ObjectMetadata getObjectMetadata(String key) throws IOException {\n        try {\n            return s3a.getObjectMetadata(new Path('/' + key));\n        } catch (SdkBaseException e) {\n            throw S3AUtils.translateException(\"getObjectMetadata\", key, e);\n        }\n    }","sourceCodeStart":112,"sourceCodeEnd":148,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/HadoopS3AccessHelper.java#L112-L148","documentation":"HadoopS3AccessHelper.copyLastPartToTargetAndSeek downloads the trailing data chunk of an interrupted multipart upload during writer recovery, then sanity-checks that the number of bytes read equals the recorded local temp-file length. A mismatch means the S3 object's last part no longer matches the local metadata recorded in the recoverable state, so appending would corrupt the output.","triggerScenarios":"Recovering an S3RecoverableWriter for resume when the last uploaded part in S3 has a different size than targetLocation.length() recorded at snapshot time — e.g. the object was modified, overwritten, lifecycle-trimmed, or the temp file metadata is stale.","commonSituations":"Multipart upload resumed after the underlying S3 object or its multipart upload parts were altered by another process; the local temp file referenced by the recoverable was truncated or recreated; clock/part-numbering drift after an aborted upload retry; or concurrent writers targeting the same key.","solutions":["Ensure only one writer at a time works on the same output key and the same local tmp directory; concurrent resumes corrupt length bookkeeping.","Verify the local temp file referenced by the recoverable still exists with its original length and was not touched by cleanup jobs.","Check S3 bucket lifecycle rules or external tools are not mutating in-progress multipart upload parts.","If the state is irrecoverable, abandon the resume, delete the partial upload (AbortMultipartUpload), and rewrite the file from scratch."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// before attempting resume, verify local temp file still matches recorded length\nFile tmp = new File(recoverable.tempFile());\nif (!tmp.exists() || tmp.length() != recoverable.offset()) {\n    // refuse to resume; the sanity check in HadoopS3AccessHelper would fail\n    throw new IOException(\"Local temp file drift detected; refusing resume\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    writer.recover(recoverable);\n} catch (IOException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"incorrect length\")) {\n        // state mismatch: abort the multipart upload and rewrite from scratch\n    } else {\n        throw e;\n    }\n}","preventionTips":["Ensure a single writer per output key; never resume the same recoverable from two tasks.","Protect local tmp directories from cleanup agents and disk-full conditions.","Prevent external processes from mutating in-progress multipart upload parts."],"tags":["s3","multipart-upload","recovery","data-integrity"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}