apache/beam · error · BeamIOError

Delete operation failed

Error message

Delete operation failed

What it means

Raised by S3FileSystem.delete when one or more of the requested paths failed to be deleted on S3. BeamIOError carries a {path: error} map of the failures; paths that deleted successfully are not reported.

Solutions

1. Inspect BeamIOError.exception_details to see which paths failed and why
2. Ensure IAM permissions include s3:DeleteObject (and version variants for versioned buckets)
3. Pre-check paths with exists() and tolerate already-deleted paths
4. Retry the failed subset after fixing permissions/rate limits

Example fix

// before
fs.delete(stale_paths)
// after
from apache_beam.io.filesystem import BeamIOError
try:
    fs.delete(stale_paths)
except BeamIOError as e:
    failed = list((e.exception_details or {}).keys())
    logging.error('failed deletions: %s', failed)

try:
    fs.delete(paths)
except BeamIOError as e:
    failed = (e.exception_details or {}).keys()
    schedule_retry(list(failed))

When it happens

Trigger: Calling fs.delete(paths) where any path is missing, is a non-empty prefix lacking ListBucket/DeleteObject permission, or S3 returns errors (throttling, AccessDenied).

Common situations: Cleanup jobs racing with writers re-creating objects; IAM policy missing s3:DeleteObject or s3:DeleteObjectVersion; deleting versioned objects without version permissions.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• Basepath %r must be S3 path.
• Checksum operation failed
• ENOENT
• error listing object keys
• exists() operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/10fd0fe94c53bc7d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/s3filesystem.py:317

      return FileMetadata(
          path, file_metadata['size'], file_metadata['last_updated'])
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("Metadata operation failed", {path: e})

  def delete(self, paths):
    """Deletes files or directories at the provided paths.
    Directories will be deleted recursively.

    Args:
      paths: list of paths that give the file objects to be deleted
    """
    results = s3io.S3IO(options=self._options).delete_paths(paths)
    exceptions = {
        path: error
        for (path, error) in results.items() if error is not None
    }
    if exceptions:
      raise BeamIOError("Delete operation failed", exceptions)

  def report_lineage(self, path, lineage):
    try:
      components = s3io.parse_s3_path(path, object_optional=True)
    except ValueError:
      # report lineage is fail-safe
      traceback.print_exc()
      return
    if components and not components[-1]:
      components = components[:-1]
    lineage.add('s3', *components, last_segment_sep='/')

View on GitHub (pinned to 12126d8942)