apache/flink · error · IOException

Failed to delete

Error message

Failed to delete: {}

What it means

Wraps an AWS SDK S3Exception raised while deleting an object (or while listing during recursive delete) in IOException('Failed to delete: <path>') with the original exception as cause. The S3 status code and error message in the cause explain the actual failure (403 AccessDenied, 404 NoSuchKey, 500s, etc.).

Solutions

1. Read the cause's status code: 403 -> grant s3:DeleteObject on the bucket/prefix; 500/503 -> retry with backoff.
2. For Object Lock-protected objects, skip deletion or use compliant retention handling; they cannot be deleted.
3. Retry the whole delete(path, true) — it is idempotent since already-deleted children simply no longer exist.

try {
    s3Fs.delete(path, true);
} catch (IOException e) {
    if (e.getCause() instanceof S3Exception) {
        int code = ((S3Exception) e.getCause()).statusCode();
        if (code == 403) {
            // fix IAM s3:DeleteObject permission, then retry
        } else if (code >= 500) {
            // transient S3 failure: retry delete with backoff (idempotent)
        } else {
            throw e;
        }
    } else {
        throw e;
    }
}

When it happens

Trigger: s3Client.deleteObject fails due to missing s3:DeleteObject permissions, object locked (Object Lock / retention), bucket owner enforcement, or transient S3 5xx errors; also fired from the recursive branch for any child delete failure.

Common situations: IAM policies grant read but not delete; S3 Object Lock or legal hold prevents deletion; deleting objects written by a different AWS account without proper ACLs/ownership; intermittent S3 errors under load.

Related errors

• Failed to rename to
• Cannot create local tmp dir
• Directory does not exist or an I/O error occurred
• Directory not empty and recursive = false
• Failed to get metadata for key

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/bfbc457eba0a9b98. Report an issue: GitHub.

Appendix: source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3FileSystem.java:405

                s3Client.deleteObject(request);
                return true;
            } else {
                if (!recursive) {
                    throw new IOException("Directory not empty and recursive = false");
                }

                final FileStatus[] contents = listStatus(path);
                for (FileStatus file : contents) {
                    delete(file.getPath(), true);
                }

                return true;
            }
        } catch (FileNotFoundException e) {
            return false;
        } catch (S3Exception e) {
            throw new IOException("Failed to delete: " + path, e);
        }
    }

    /**
     * Creates a directory at the specified path.
     *
     * <p><b>S3 Behavior:</b> S3 is a flat object store and doesn't have true directories. Directory
     * semantics are simulated through key prefixes. This method always returns true because:
     *
     * <ul>
     *   <li>S3 doesn't require directories to exist before creating objects with that prefix
     *   <li>Creating an empty "directory marker" object (key ending with /) is optional
     *   <li>Most S3 implementations don't create these markers for consistency with Hadoop FS
     * </ul>
     *
     * <p>If explicit directory markers are needed, consider using a custom implementation.
     *
     * @return always returns true (S3 doesn't require explicit directory creation)

View on GitHub (pinned to 2f3c205e92)