{"record":{"id":"e8d854fd8d34b7a6","repo":"apache/flink","slug":"incomplete-tail-object-has-unexpected-length-e","errorCode":null,"errorMessage":"Incomplete-tail object {} has unexpected length (expected {} bytes, got {} bytes). The side object holding the in-flight tail has been truncated, overwritten, or replaced out-of-band since the checkpoint was taken. Recovery cannot proceed: the writer state is inconsistent with the checkpoint and this failure is NOT retriable from the same checkpoint. Either restore the side object to its original length or roll back to an earlier checkpoint that does not reference it.","messagePattern":"Incomplete-tail object (.+?) has unexpected length \\(expected (.+?) bytes, got (.+?) bytes\\)\\. The side object holding the in-flight tail has been truncated, overwritten, or replaced out-of-band since the checkpoint was taken\\. Recovery cannot proceed: the writer state is inconsistent with the checkpoint and this failure is NOT retriable from the same checkpoint\\. Either restore the side object to its original length or roll back to an earlier checkpoint that does not reference it\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3RecoverableWriter.java","lineNumber":157,"sourceCode":"\n    /**\n     * Downloads the side object holding the previously-persisted sub-part-size tail into a fresh\n     * file under {@link #localTmpDir}. The side object itself is left in place so that a repeated\n     * recovery from the same checkpoint remains correct; cleanup is the responsibility of {@link\n     * #cleanupRecoverableState(ResumeRecoverable)} which Flink invokes when the checkpoint is\n     * retired.\n     */\n    private File downloadIncompleteTail(NativeS3Recoverable s3recoverable) throws IOException {\n        final File tmpDir = new File(localTmpDir);\n        if (!tmpDir.exists() && !tmpDir.mkdirs()) {\n            throw new IOException(\"Cannot create local tmp dir: \" + localTmpDir);\n        }\n        final File target = new File(tmpDir, \"s3-resume-\" + UUID.randomUUID());\n        try {\n            final long downloaded =\n                    s3AccessHelper.getObject(s3recoverable.incompleteObjectName(), target);\n            if (downloaded != s3recoverable.incompleteObjectLength()) {\n                throw new IOException(\n                        \"Incomplete-tail object \"\n                                + s3recoverable.incompleteObjectName()\n                                + \" has unexpected length (expected \"\n                                + s3recoverable.incompleteObjectLength()\n                                + \" bytes, got \"\n                                + downloaded\n                                + \" bytes). The side object holding the in-flight tail \"\n                                + \"has been truncated, overwritten, or replaced out-of-band \"\n                                + \"since the checkpoint was taken. Recovery cannot proceed: \"\n                                + \"the writer state is inconsistent with the checkpoint and \"\n                                + \"this failure is NOT retriable from the same checkpoint. \"\n                                + \"Either restore the side object to its original length or \"\n                                + \"roll back to an earlier checkpoint that does not reference it.\");\n            }\n            return target;\n        } catch (IOException e) {\n            try {\n                Files.deleteIfExists(target.toPath());","sourceCodeStart":139,"sourceCodeEnd":175,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3RecoverableWriter.java#L139-L175","documentation":"downloadIncompleteTail compares the downloaded side-object byte count against the length recorded in the NativeS3Recoverable; a mismatch means the S3 object holding the in-flight tail changed out-of-band (truncated, overwritten, replaced) since the checkpoint. The message is explicit: the checkpoint's writer state is inconsistent with S3 reality and recovery from that checkpoint is not retriable.","triggerScenarios":"recover() where getObject succeeds but the object's size != incompleteObjectLength(): S3 lifecycle/retention policies rewrote or truncated the side object, another process overwrote the same key, or the same side-object key was reused by a concurrent job.","commonSituations":"S3 lifecycle rules that transition/modify objects in the prefix used for recoverable state; two jobs writing to identical bucket/prefix producing key collisions; bucket used for checkpoints also targeted by compaction/ETL jobs; versioning enabled with noncurrent-object interference.","solutions":["Restore the side object to its checkpointed length (e.g. from S3 object versioning: retrieve the prior version) or roll back to an earlier checkpoint that does not reference it — as the message instructs.","Give each job a distinct, dedicated bucket/prefix for S3 recoverable state to eliminate key collisions.","Audit S3 lifecycle rules on the checkpoint bucket and exclude the recoverable-state prefix from any transition/expiration for the job's lifetime.","Prevent out-of-band writers (scripts, other jobs) from touching the checkpoint/recoverable prefix."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    stream = writer.recover(rec);\n} catch (IOException e) {\n    if (e.getMessage().contains(\"unexpected length\")) {\n        // side object mutated out-of-band: restore prior checkpoint or\n        // retrieve prior S3 object version and retry\n    }\n}","preventionTips":["Enable S3 versioning on the checkpoint bucket for rollback.","Use dedicated, untouched prefixes for recoverable state.","Audit lifecycle rules against the checkpoint prefix."],"tags":["s3","recovery","state-corruption","lifecycle-rules","checkpoint"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}