apache/beam · error · BeamIOError

Delete operation failed

Error message

Delete operation failed

What it means

Filesystem.delete() collects per-path failures (each as BeamIOError) in an exceptions dict; if any path failed to delete it raises a single BeamIOError('Delete operation failed') whose exception_payload carries the per-path details. It also raises if no files matched the pattern ('No files found to delete under: ...').

Solutions

1. Inspect the exception_payload of the raised BeamIOError to find which specific paths failed, then fix those individually.
2. Verify the glob/path list matches existing files before deleting (use match() first).
3. Fix permissions or release file locks for the failing paths and retry.
4. Wrap delete() in try/except BeamIOError and treat per-path failures as non-fatal if cleanup is best-effort.

Example fix

// before
fs.delete(['gs-tmp-*'])
// after
try:
    fs.delete(['/tmp/output/*.tmp'])
except BeamIOError as e:
    for path, err in (e.exception_payload or {}).items():
        print('failed to delete', path, err)

matches = fs.match([pattern])
assert matches[0].metadata_list, 'nothing to delete'

try:
    fs.delete(paths)
except BeamIOError as e:
    failed = getattr(e, 'exception_payload', {})
    # retry or skip only the failed paths

When it happens

Trigger: Passing a list of paths/globs to delete() where one or more paths cannot be removed (permissions, open file handles, vanished mid-run), or where the glob matches no files at all.

Common situations: Batch cleanup of Beam temp/staging files where a running job still holds files; glob patterns with a typo matching nothing; mixed-path batches where a subset of deletions fail.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• error getting last modified time for
• errorForMissingScheme(scheme, path)
• no files matching pattern
• rethrown stored IOException (typically "MatchResult status…
• apache_beam.io.gcp.datastore.v1new.datastoreio.Entity…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5c76b0f176e6ae00. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/localfilesystem.py:368

    def try_delete(path):
      try:
        _delete_path(path)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[path] = e

    for match_result in self.match(paths):
      metadata_list = match_result.metadata_list

      if not metadata_list:
        exceptions[match_result.pattern] = \
          IOError('No files found to delete under: %s' % match_result.pattern)

      for metadata in match_result.metadata_list:
        try_delete(metadata.path)

    if exceptions:
      raise BeamIOError("Delete operation failed", exceptions)

  def report_lineage(self, path, lineage):
    lineage.add('filesystem', 'localhost', path, last_segment_sep='/')

View on GitHub (pinned to 12126d8942)