apache/flink · critical · IOException

Incomplete-tail object {} has unexpected length (expected {}

Error message

Incomplete-tail object {} has unexpected length (expected {} bytes, got {} bytes). The side object holding the in-flight tail has been truncated, overwritten, or replaced out-of-band since the checkpoint was taken. Recovery cannot proceed: the writer state is inconsistent with the checkpoint and this failure is NOT retriable from the same checkpoint. Either restore the side object to its original length or roll back to an earlier checkpoint that does not reference it.

What it means

downloadIncompleteTail compares the downloaded side-object byte count against the length recorded in the NativeS3Recoverable; a mismatch means the S3 object holding the in-flight tail changed out-of-band (truncated, overwritten, replaced) since the checkpoint. The message is explicit: the checkpoint's writer state is inconsistent with S3 reality and recovery from that checkpoint is not retriable.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3RecoverableWriter.java:157

    /**
     * Downloads the side object holding the previously-persisted sub-part-size tail into a fresh
     * file under {@link #localTmpDir}. The side object itself is left in place so that a repeated
     * recovery from the same checkpoint remains correct; cleanup is the responsibility of {@link
     * #cleanupRecoverableState(ResumeRecoverable)} which Flink invokes when the checkpoint is
     * retired.
     */
    private File downloadIncompleteTail(NativeS3Recoverable s3recoverable) throws IOException {
        final File tmpDir = new File(localTmpDir);
        if (!tmpDir.exists() && !tmpDir.mkdirs()) {
            throw new IOException("Cannot create local tmp dir: " + localTmpDir);
        }
        final File target = new File(tmpDir, "s3-resume-" + UUID.randomUUID());
        try {
            final long downloaded =
                    s3AccessHelper.getObject(s3recoverable.incompleteObjectName(), target);
            if (downloaded != s3recoverable.incompleteObjectLength()) {
                throw new IOException(
                        "Incomplete-tail object "
                                + s3recoverable.incompleteObjectName()
                                + " has unexpected length (expected "
                                + s3recoverable.incompleteObjectLength()
                                + " bytes, got "
                                + downloaded
                                + " bytes). The side object holding the in-flight tail "
                                + "has been truncated, overwritten, or replaced out-of-band "
                                + "since the checkpoint was taken. Recovery cannot proceed: "
                                + "the writer state is inconsistent with the checkpoint and "
                                + "this failure is NOT retriable from the same checkpoint. "
                                + "Either restore the side object to its original length or "
                                + "roll back to an earlier checkpoint that does not reference it.");
            }
            return target;
        } catch (IOException e) {
            try {
                Files.deleteIfExists(target.toPath());

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Restore the side object to its checkpointed length (e.g. from S3 object versioning: retrieve the prior version) or roll back to an earlier checkpoint that does not reference it — as the message instructs.
2. Give each job a distinct, dedicated bucket/prefix for S3 recoverable state to eliminate key collisions.
3. Audit S3 lifecycle rules on the checkpoint bucket and exclude the recoverable-state prefix from any transition/expiration for the job's lifetime.
4. Prevent out-of-band writers (scripts, other jobs) from touching the checkpoint/recoverable prefix.

try {
    stream = writer.recover(rec);
} catch (IOException e) {
    if (e.getMessage().contains("unexpected length")) {
        // side object mutated out-of-band: restore prior checkpoint or
        // retrieve prior S3 object version and retry
    }
}

When it happens

Trigger: recover() where getObject succeeds but the object's size != incompleteObjectLength(): S3 lifecycle/retention policies rewrote or truncated the side object, another process overwrote the same key, or the same side-object key was reused by a concurrent job.

Common situations: S3 lifecycle rules that transition/modify objects in the prefix used for recoverable state; two jobs writing to identical bucket/prefix producing key collisions; bucket used for checkpoints also targeted by compaction/ETL jobs; versioning enabled with noncurrent-object interference.

Related errors

• Incomplete-tail object {} is referenced with non-positive le
• Failed to complete multipart upload for key: {}
• Error recovering writer: Downloading the last data chunk fil
• Failed to get object for key: {}
• Incomplete-tail file does not exist: {}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/e8d854fd8d34b7a6. Report an issue: GitHub.