apache/flink · error · IOException

Cannot create local tmp dir: {}

Error message

Cannot create local tmp dir: {}

What it means

Thrown by NativeS3RecoverableWriter.downloadIncompleteTail when the local tmp directory neither exists nor can be created with mkdirs(). Recovery needs scratch space to download the in-flight tail object; without it, resume cannot proceed.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3RecoverableWriter.java:150

                } catch (IOException cleanup) {
                    e.addSuppressed(cleanup);
                }
            }
            throw e;
        }
    }

    /**
     * Downloads the side object holding the previously-persisted sub-part-size tail into a fresh
     * file under {@link #localTmpDir}. The side object itself is left in place so that a repeated
     * recovery from the same checkpoint remains correct; cleanup is the responsibility of {@link
     * #cleanupRecoverableState(ResumeRecoverable)} which Flink invokes when the checkpoint is
     * retired.
     */
    private File downloadIncompleteTail(NativeS3Recoverable s3recoverable) throws IOException {
        final File tmpDir = new File(localTmpDir);
        if (!tmpDir.exists() && !tmpDir.mkdirs()) {
            throw new IOException("Cannot create local tmp dir: " + localTmpDir);
        }
        final File target = new File(tmpDir, "s3-resume-" + UUID.randomUUID());
        try {
            final long downloaded =
                    s3AccessHelper.getObject(s3recoverable.incompleteObjectName(), target);
            if (downloaded != s3recoverable.incompleteObjectLength()) {
                throw new IOException(
                        "Incomplete-tail object "
                                + s3recoverable.incompleteObjectName()
                                + " has unexpected length (expected "
                                + s3recoverable.incompleteObjectLength()
                                + " bytes, got "
                                + downloaded
                                + " bytes). The side object holding the in-flight tail "
                                + "has been truncated, overwritten, or replaced out-of-band "
                                + "since the checkpoint was taken. Recovery cannot proceed: "
                                + "the writer state is inconsistent with the checkpoint and "
                                + "this failure is NOT retriable from the same checkpoint. "

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Check the configured local temporary directory (S3 filesystem config) and fix ownership/permissions so the Flink process can create it.
2. Point the tmp dir at writable, adequately-sized local storage (not read-only mounts); prefer a dedicated volume on TaskManagers.
3. Free disk space or enlarge the volume if full — multipart tails plus part buffering need headroom.
4. Verify with a manual test on the node: sudo -u <flink-user> mkdir -p <localTmpDir>.

File d = new File(localTmpDir);
if (!d.exists() && !d.mkdirs()) {
    throw new IllegalStateException("tmp dir not creatable: " + localTmpDir);
}

When it happens

Trigger: recover() with a non-null incomplete object when localTmpDir is unwritable (permissions), its parent is a file, the disk is full, or the path is invalid on the filesystem (e.g. embedded NUL, or a read-only mount). mkdirs() returning false covers all non-exception failure modes.

Common situations: TaskManager running as a user without write permission to the configured tmp dir; container with read-only root FS and tmp dir pointing inside it; disk exhaustion; localTmpDir defaulting into a path that another job holds as a regular file; SELinux/AppArmor denying writes.

Related errors

• Native S3 File System cannot recover recoverable for other f
• Unable to recover. Rename operation failed
• Error recovering writer: Downloading the last data chunk fil
• Failed to delete: {}
• Failed to rename {} to {}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/cfbdc82c43d71204. Report an issue: GitHub.