{"record":{"id":"aae2709e740491da","repo":"apache/flink","slug":"incomplete-tail-object-is-referenced-with-non-p","errorCode":null,"errorMessage":"Incomplete-tail object {} is referenced with non-positive length {}. A side object is only written when buffered tail bytes exist, so this indicates corrupt recoverable metadata. Recovery cannot proceed and this failure is NOT retriable from the same checkpoint.","messagePattern":"Incomplete-tail object (.+?) is referenced with non-positive length (.+?)\\. A side object is only written when buffered tail bytes exist, so this indicates corrupt recoverable metadata\\. Recovery cannot proceed and this failure is NOT retriable from the same checkpoint\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3RecoverableWriter.java","lineNumber":95,"sourceCode":"    }\n\n    @Override\n    public RecoverableFsDataOutputStream.Committer recoverForCommit(CommitRecoverable recoverable)\n            throws IOException {\n        checkNotClosed();\n        NativeS3Recoverable s3recoverable = castToNativeS3Recoverable(recoverable);\n        return new NativeS3Committer(s3AccessHelper, s3recoverable);\n    }\n\n    @Override\n    public RecoverableFsDataOutputStream recover(ResumeRecoverable recoverable) throws IOException {\n        checkNotClosed();\n        final NativeS3Recoverable s3recoverable = castToNativeS3Recoverable(recoverable);\n\n        File incompleteTail = null;\n        if (s3recoverable.incompleteObjectName() != null) {\n            if (s3recoverable.incompleteObjectLength() <= 0) {\n                throw new IOException(\n                        \"Incomplete-tail object \"\n                                + s3recoverable.incompleteObjectName()\n                                + \" is referenced with non-positive length \"\n                                + s3recoverable.incompleteObjectLength()\n                                + \". A side object is only written when buffered tail bytes \"\n                                + \"exist, so this indicates corrupt recoverable metadata. \"\n                                + \"Recovery cannot proceed and this failure is NOT retriable \"\n                                + \"from the same checkpoint.\");\n            }\n            incompleteTail = downloadIncompleteTail(s3recoverable);\n        }\n\n        try {\n            LOG.debug(\n                    \"Resuming stream - key: {}, uploadId: {}, parts: {}, bytesInParts: {}, incompleteTail: {} ({} bytes)\",\n                    s3recoverable.getObjectName(),\n                    s3recoverable.uploadId(),\n                    s3recoverable.parts().size(),","sourceCodeStart":77,"sourceCodeEnd":113,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3RecoverableWriter.java#L77-L113","documentation":"A defensive invariant check in NativeS3RecoverableWriter.recover: the recoverable names an incomplete-tail side object but records length <= 0. The writer only ever persists a side object when buffered tail bytes exist (length > 0), so this combination cannot come from a healthy writer — it signals corrupt recoverable metadata, and the message explicitly states recovery from that checkpoint is not retriable.","triggerScenarios":"recover(ResumeRecoverable) where incompleteObjectName() != null and incompleteObjectLength() <= 0 — e.g. deserialized NativeS3Recoverable from a checkpoint whose bytes were corrupted, a hand-built/modified recoverable, or a bug in serializer round-trip.","commonSituations":"Bit-rot or truncation in state backend bytes; a serializer version mismatch partially decoded (see the separate version check); state migrated between backends with lossy tooling; programmatic construction of NativeS3Recoverable with a name but zero length.","solutions":["Do not retry from the same checkpoint — as the message says, it will fail identically; roll back to an earlier checkpoint/savepoint.","Inspect how the state was produced: confirm the writing job used the same plugin version and a healthy state backend.","If state was copied/migrated (savepoint tooling, state processor API), verify the migration preserved recoverable bytes exactly.","Report as a bug if reproducible with a clean checkpoint — a healthy writer should never emit this shape."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"if (rec.incompleteObjectName() != null && rec.incompleteObjectLength() <= 0) {\n    // corrupt metadata — skip this checkpoint, restore an earlier one\n}","typeGuard":null,"tryCatchPattern":"try {\n    stream = writer.recover(rec);\n} catch (IOException e) {\n    if (e.getMessage().contains(\"non-positive length\")) {\n        // fall back to previous checkpoint\n    }\n}","preventionTips":["Use a durable, checksummed state backend.","Do not hand-edit or hand-build recoverable objects.","Keep plugin versions consistent to avoid malformed state."],"tags":["s3","recovery","state-corruption","checkpoint","invariant"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}