apache/flink · error · IOException

Incomplete-tail file does not exist: {}

Error message

Incomplete-tail file does not exist: {}

What it means

Thrown by NativeS3RecoverableFsDataOutputStream.resumeFromIncompleteTail when the local file passed to resume the stream does not exist on disk. The constructor expects the previously downloaded tail file to be present; its absence means local state was lost between download and stream construction.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3RecoverableFsDataOutputStream.java:121

        this.uploadId = uploadId;
        this.localTmpDir = localTmpDir;
        this.minPartSize = minPartSize;
        this.completedParts = new ArrayList<>(existingParts);
        this.numBytesInParts = numBytesInParts;
        this.nextPartNumber = existingParts.size() + 1;
        this.currentPartSize = 0;
        this.closed = false;

        if (incompleteTailFile != null) {
            resumeFromIncompleteTail(incompleteTailFile);
        } else {
            createNewTempFile();
        }
    }

    private void resumeFromIncompleteTail(File tailFile) throws IOException {
        if (!tailFile.exists()) {
            throw new IOException("Incomplete-tail file does not exist: " + tailFile);
        }
        currentTempFile = tailFile;
        currentPartSize = tailFile.length();
        // Append mode so subsequent writes land after the recovered bytes.
        currentFileStream = new FileOutputStream(currentTempFile, true);
        currentOutputStream = new BufferedOutputStream(currentFileStream, BUFFER_SIZE);
    }

    private void createNewTempFile() throws IOException {
        File tmpDir = new File(localTmpDir);
        Files.createDirectories(tmpDir.toPath());

        currentTempFile = new File(tmpDir, "s3-part-" + UUID.randomUUID());
        currentFileStream = new FileOutputStream(currentTempFile);
        currentOutputStream = new BufferedOutputStream(currentFileStream, BUFFER_SIZE);
        currentPartSize = 0;
    }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Recover on the same TaskManager/local directory that holds the tail file, or re-download the incomplete tail via NativeS3RecoverableWriter.recover() which re-fetches from S3 instead of reusing a stale local path.
2. Configure localTmpDir on durable storage that survives task restarts (not OS-managed /tmp with tmpfs/cleanup).
3. Check that no external process (systemd-tmpfiles, container reaper) purges the tmp dir during job lifetime.
4. If the local file is unrecoverable, restart from the checkpoint using the standard recover() path rather than manually constructing the stream.

Example fix

// before
new NativeS3RecoverableFsDataOutputStream(..., tailFile); // file may be gone

// after
if (!tailFile.exists()) {
    // re-download from S3 via the writer instead of assuming the local copy
    tailFile = writerRedownloadPath; // NativeS3RecoverableWriter.recover(recoverable)
}

if (incompleteTailFile != null && !incompleteTailFile.exists()) {
    // re-download via NativeS3RecoverableWriter.recover(recoverable) instead of
    // constructing the stream with the stale local path
}

When it happens

Trigger: Constructing the recoverable output stream with a non-null incompleteTailFile (typically produced by NativeS3RecoverableWriter.downloadIncompleteTail) whose path no longer resolves to an existing file — e.g. localTmpDir was wiped, the file was deleted by another process, or the path was built for a different node after a TaskManager failover.

Common situations: TaskManager restart where local tmp dirs (/tmp) are cleaned between attempts; localTmpDir on ephemeral container storage; recovery happening on a different physical machine than where the tail was downloaded; antivirus/cleanup jobs deleting files in tmp; disk-full followed by cleanup scripts.

Related errors

• Failed to get object for key: {}
• Failed to get metadata for key: {}
• Native S3 File System cannot recover recoverable for other f
• An I/O error occurred while creating temporary file to extra
• Failed to create parent(s) for given base dir: %s

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/43c80c8507ec40a1. Report an issue: GitHub.