apache/beam · error · BeamIOError

Delete operation failed

Error message

Delete operation failed

What it means

GcsFileSystem.delete raises BeamIOError('Delete operation failed') when any of the requested deletions fails. It matches paths, then deletes each; per-target exceptions are aggregated into a single BeamIOError keyed by path.

Solutions

1. Read exception_details to identify which paths failed and why.
2. Grant the service account storage.objects.delete permission on the bucket.
3. Check existence (exists/match) and skip already-deleted paths.
4. Retry the failed subset from exception_details.

Example fix

// before
fs.delete(paths)
// after
try:
  fs.delete(paths)
except BeamIOError as e:
  for target, err in e.exception_details.items():
    logging.warning('failed to delete %s: %s', target, err)

targets = [m.path for m in fs.match([p if '*' in p else p + '*'])[0].metadata_list]

def valid_gcs_path(p):
    return isinstance(p, str) and p.startswith('gs://') and p.count('/') >= 3

try:
    fs.delete(paths)
except BeamIOError as e:
    for target, err in e.exception_details.items():
        logging.warning('delete %s failed: %s', target, err)

When it happens

Trigger: Calling GcsFileSystem.delete(paths) where at least one path cannot be deleted: object missing, permission denied on the bucket, or a matched target fails during deletion.

Common situations: Cleaning up temp files already removed by another worker; service account lacking storage.objects.delete; deleting prefixes with thousands of objects where some fail.

Related errors

• Copy operation failed
• Error trying to delete
• Rename operation failed
• Checksum operation failed
• Delete operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a79bb3354e4fca05. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/gcsfilesystem.py:369

    """

    exceptions = {}

    for path in paths:
      if path.endswith('/'):
        self._gcsIO().delete(path, recursive=True)
        continue
      else:
        path_to_use = path
      match_result = self.match([path_to_use])[0]
      statuses = self._gcsIO().delete_batch(
          [m.path for m in match_result.metadata_list])
      for target, exception in statuses:
        if exception:
          exceptions[target] = exception

    if exceptions:
      raise BeamIOError("Delete operation failed", exceptions)

  def report_lineage(self, path, lineage):
    try:
      components = gcsio.parse_gcs_path(path, object_optional=True)
    except ValueError:
      # report lineage is fail-safe
      traceback.print_exc()
      return
    if components and not components[-1]:
      components = components[:-1]
    lineage.add('gcs', *components, last_segment_sep='/')

View on GitHub (pinned to 12126d8942)