apache/flink · warning · FileNotFoundException

{directory}

Error message

{directory}

What it means

A FileNotFoundException whose message is just the directory path, thrown from cleanDirectoryInternal when listFiles() returned null and exists() then reported the directory as gone. It means the directory disappeared between the two checks — a benign race with a concurrent deleter, surfaced as an exception.

Source

Thrown at flink-core/src/main/java/org/apache/flink/util/FileUtils.java:356

            throw new IOException(directory + " is not a directory");
        }
        // else: does not exist, which is okay (as if deleted)
    }

    private static void cleanDirectoryInternal(File directory) throws IOException {
        if (Files.isSymbolicLink(directory.toPath())) {
            // the user directories which symbolic links point to should not be cleaned.
            return;
        }
        if (directory.isDirectory()) {
            final File[] files = directory.listFiles();

            if (files == null) {
                // directory does not exist any more or no permissions
                if (directory.exists()) {
                    throw new IOException("Failed to list contents of " + directory);
                } else {
                    throw new FileNotFoundException(directory.toString());
                }
            }

            // remove all files in the directory
            for (File file : files) {
                if (file != null) {
                    deleteFileOrDirectory(file);
                }
            }
        } else if (directory.exists()) {
            throw new IOException(directory + " is not a directory but a regular file");
        } else {
            // else does not exist at all
            throw new FileNotFoundException(directory.toString());
        }
    }

    private static void guardIfNotThreadSafe(ThrowingConsumer<File, IOException> toRun, File file)

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Serialize cleanup of shared directories (single owner, lock, or sequentialize teardown) so two cleaners never race
2. Catch FileNotFoundException around deleteDirectory when the race is benign and idempotent deletion is intended
3. Give each component its own working directory so cleanups cannot collide

Example fix

// before
FileUtils.deleteDirectory(sharedWorkDir); // races with other cleaner

// after
try {
    FileUtils.deleteDirectory(sharedWorkDir);
} catch (FileNotFoundException e) {
    // already removed concurrently; deletion is idempotent for us
    LOG.debug("Directory already deleted concurrently: {}", sharedWorkDir);
}

try { FileUtils.deleteDirectory(dir); } catch (FileNotFoundException e) { /* removed concurrently — treat as success */ }

When it happens

Trigger: Two threads/processes concurrently cleaning the same directory: listFiles() fails because the dir is being removed, and by the time exists() runs it is already deleted. Common when a shutdown hook and a supervisor both wipe the same work/temp dir.

Common situations: Race between task failure recovery cleanup and JobManager-local cleanup; test teardown running concurrently with async cleanup from the previous test; external janitor scripts deleting the same temp dirs.

Related errors

• {directory} is not a directory
• Failed to list contents of {directory}
• {directory} is not a directory but a regular file
• DistributedCache supports only local files for Collection En
• Concurrent access to KryoSerializer. Thread 1: {threadName1}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/aeb96765ff2ad827. Report an issue: GitHub.