What causes "Directory not empty and recursive = false" in apache/flink?

Pass recursive=true when deleting directories: s3Fs.delete(dirPath, true).

apache/flink · error · IOException

Directory not empty and recursive = false

Error message

Directory not empty and recursive = false

What it means

NativeS3FileSystem.delete throws IOException('Directory not empty and recursive = false') when delete is called on a path that getFileStatus reports as a directory and recursive=false. Note the check fires for ANY directory delete without the recursive flag, even if the directory prefix currently has no children, because directory-ness is inferred from listing and non-recursive directory deletes are simply not supported.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3FileSystem.java:392

    @Override
    public boolean delete(Path path, boolean recursive) throws IOException {
        checkNotClosed();
        final String key = NativeS3ObjectOperations.extractKey(path);
        final S3Client s3Client = clientProvider.getS3Client();

        try {
            final FileStatus status = getFileStatus(path);

            if (!status.isDir()) {
                final DeleteObjectRequest request =
                        DeleteObjectRequest.builder().bucket(bucketName).key(key).build();

                s3Client.deleteObject(request);
                return true;
            } else {
                if (!recursive) {
                    throw new IOException("Directory not empty and recursive = false");
                }

                final FileStatus[] contents = listStatus(path);
                for (FileStatus file : contents) {
                    delete(file.getPath(), true);
                }

                return true;
            }
        } catch (FileNotFoundException e) {
            return false;
        } catch (S3Exception e) {
            throw new IOException("Failed to delete: " + path, e);
        }
    }

    /**
     * Creates a directory at the specified path.

View on GitHub (pinned to 2f3c205e92)

Solutions

Pass recursive=true when deleting directories: s3Fs.delete(dirPath, true).
If you need non-recursive semantics, first listStatus(path) and delete children individually, then handle the directory itself.
Guard the call: only call delete(...,false) when !status.isDir().

Example fix

// before
boolean deleted = s3Fs.delete(dirPath, false); // IOException

// after
FileStatus st = s3Fs.getFileStatus(dirPath);
boolean deleted = s3Fs.delete(dirPath, /*recursive=*/ st.isDir());

Defensive patterns

Strategy: validation

Validate before calling

// choose recursive flag from the actual status
FileStatus st = s3Fs.getFileStatus(path);
boolean recursive = st.isDir(); // non-recursive delete only valid for files
boolean ok = s3Fs.delete(path, recursive);

Try / catch

try {
    s3Fs.delete(path, false);
} catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("recursive = false")) {
        s3Fs.delete(path, true); // retry recursively if that is acceptable
    } else {
        throw e;
    }
}

Prevention

Do not port POSIX rmdir semantics to S3; directories here are prefixes.
Pass recursive=true whenever the path may be a directory.
Delete children explicitly via listStatus when you need selective cleanup.

When it happens

Trigger: Calling s3Fs.delete(dirPath, false) where dirPath resolves to a directory FileStatus (objects exist under the prefix, or a directory marker exists).

Common situations: Cleanup code ported from a POSIX-style filesystem that calls delete(path, false) on directories; FileOutputCommitter-style task cleanup deleting output directories non-recursively; expecting rmdir semantics on S3.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/b21135092f998aff. Report an issue: GitHub.