apache/beam · error · BeamIOError

Delete operation failed

Error message

Delete operation failed

What it means

BlobStorageFileSystem.delete(paths) deletes files/directories recursively via BlobStorageIO.delete_files() and raises BeamIOError('Delete operation failed', exceptions) when any path failed, where exceptions maps each failed path to its error. Unlike rename, there is no count check — this error only reflects remote per-path failures.

Source

Thrown at sdks/python/apache_beam/io/azure/blobstoragefilesystem.py:319

  def delete(self, paths):
    """Deletes files or directories at the provided paths.
    Directories will be deleted recursively.

    Args:
      paths: list of paths that give the file objects to be deleted

    Raises:
      ``BeamIOError``: if any of the delete operations fail
    """
    results = self._blobstorageIO().delete_paths(paths)
    # Retrieve exceptions.
    exceptions = {
        path: error
        for (path, error) in results.items() if error is not None
    }

    if exceptions:
      raise BeamIOError("Delete operation failed", exceptions)

  def report_lineage(self, path, lineage):
    try:
      components = blobstorageio.parse_azfs_path(
          path, blob_optional=True, get_account=True)
    except ValueError:
      # report lineage is fail-safe
      traceback.print_exc()
      return
    if components and not components[-1]:
      components = components[:-1]
    lineage.add('abs', *components, last_segment_sep='/')

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect BeamIOError.exception_details to see which paths failed and retry only those
2. Ensure credentials/SAS have delete permissions (Storage Blob Data Contributor or equivalent)
3. Disable or account for soft-delete/retention/lease policies on the container
4. Make deletes idempotent: treat 'not found' failures as success in cleanup code by filtering them from a retry

Example fix

// before
fs.delete(paths)
// after
try:
    fs.delete(paths)
except BeamIOError as e:
    retry = [p for p, err in e.exception_details.items() if not fs.exists(p) is False or True]
    # simpler: retry paths that still exist
    retry = [p for p in e.exception_details if fs.exists(p)]
    if retry:
        fs.delete(retry)

# filter paths that still exist and are safe to delete
existing = [p for p in paths if fs.exists(p)]

try:
    fs.delete(paths)
except BeamIOError as e:
    failed = e.exception_details  # dict {path: error}
    retry = [p for p in failed if fs.exists(p)]  # skip already-deleted
    if retry:
        fs.delete(retry)

When it happens

Trigger: Calling fs.delete(paths) where at least one path fails to delete: blob/container does not exist, credentials lack delete permission, container has a delete retention policy/lease, or transient Azure errors. Results are collected per path and any non-None error triggers the raise.

Common situations: Cleanup of temp/staging blobs that were already removed by a lifecycle management policy or another worker; service principal missing 'Storage Blob Data Contributor' delete rights; deleting containers with immutable/soft-delete policies; throttling during large batch deletes.

Related errors

• Rename operation failed.
• Unable to rename unequal number of sources and destinations.
• Exists operation failed
• Size operation failed
• Last updated operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/06eaa38c8f544334. Report an issue: GitHub.