apache/flink · critical · IOException

Incomplete-tail object {} is referenced with non-positive le

Error message

Incomplete-tail object {} is referenced with non-positive length {}. A side object is only written when buffered tail bytes exist, so this indicates corrupt recoverable metadata. Recovery cannot proceed and this failure is NOT retriable from the same checkpoint.

What it means

A defensive invariant check in NativeS3RecoverableWriter.recover: the recoverable names an incomplete-tail side object but records length <= 0. The writer only ever persists a side object when buffered tail bytes exist (length > 0), so this combination cannot come from a healthy writer — it signals corrupt recoverable metadata, and the message explicitly states recovery from that checkpoint is not retriable.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3RecoverableWriter.java:95

    }

    @Override
    public RecoverableFsDataOutputStream.Committer recoverForCommit(CommitRecoverable recoverable)
            throws IOException {
        checkNotClosed();
        NativeS3Recoverable s3recoverable = castToNativeS3Recoverable(recoverable);
        return new NativeS3Committer(s3AccessHelper, s3recoverable);
    }

    @Override
    public RecoverableFsDataOutputStream recover(ResumeRecoverable recoverable) throws IOException {
        checkNotClosed();
        final NativeS3Recoverable s3recoverable = castToNativeS3Recoverable(recoverable);

        File incompleteTail = null;
        if (s3recoverable.incompleteObjectName() != null) {
            if (s3recoverable.incompleteObjectLength() <= 0) {
                throw new IOException(
                        "Incomplete-tail object "
                                + s3recoverable.incompleteObjectName()
                                + " is referenced with non-positive length "
                                + s3recoverable.incompleteObjectLength()
                                + ". A side object is only written when buffered tail bytes "
                                + "exist, so this indicates corrupt recoverable metadata. "
                                + "Recovery cannot proceed and this failure is NOT retriable "
                                + "from the same checkpoint.");
            }
            incompleteTail = downloadIncompleteTail(s3recoverable);
        }

        try {
            LOG.debug(
                    "Resuming stream - key: {}, uploadId: {}, parts: {}, bytesInParts: {}, incompleteTail: {} ({} bytes)",
                    s3recoverable.getObjectName(),
                    s3recoverable.uploadId(),
                    s3recoverable.parts().size(),

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Do not retry from the same checkpoint — as the message says, it will fail identically; roll back to an earlier checkpoint/savepoint.
2. Inspect how the state was produced: confirm the writing job used the same plugin version and a healthy state backend.
3. If state was copied/migrated (savepoint tooling, state processor API), verify the migration preserved recoverable bytes exactly.
4. Report as a bug if reproducible with a clean checkpoint — a healthy writer should never emit this shape.

if (rec.incompleteObjectName() != null && rec.incompleteObjectLength() <= 0) {
    // corrupt metadata — skip this checkpoint, restore an earlier one
}

try {
    stream = writer.recover(rec);
} catch (IOException e) {
    if (e.getMessage().contains("non-positive length")) {
        // fall back to previous checkpoint
    }
}

When it happens

Trigger: recover(ResumeRecoverable) where incompleteObjectName() != null and incompleteObjectLength() <= 0 — e.g. deserialized NativeS3Recoverable from a checkpoint whose bytes were corrupted, a hand-built/modified recoverable, or a bug in serializer round-trip.

Common situations: Bit-rot or truncation in state backend bytes; a serializer version mismatch partially decoded (see the separate version check); state migrated between backends with lossy tooling; programmatic construction of NativeS3Recoverable with a name but zero length.

Related errors

• Incomplete-tail object {} has unexpected length (expected {}
• Error recovering writer: Downloading the last data chunk fil
• Failed to complete multipart upload for key: {}
• Failed to get object for key: {}
• Incomplete-tail file does not exist: {}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/aae2709e740491da. Report an issue: GitHub.