apache/beam · error · BeamIOError

Delete operation failed

Error message

Delete operation failed

What it means

HadoopFileSystem.delete attempts to delete each URL recursively, collecting any exception per URL; if any deletion fails it raises a single BeamIOError 'Delete operation failed' with the per-URL causes in exception_payload. Permission errors, connection errors, or URL parse failures land here.

Solutions

1. Inspect BeamIOError.exception_payload to identify which URLs failed and the underlying error.
2. Check HDFS permissions on the paths and the deleting user (hdfs dfs -chmod/-chown, or run as the file owner).
3. Verify Kerberos auth is valid (kinit) and HADOOP_CONF_DIR points to the correct cluster.
4. Skip URLs that no longer exist instead of deleting them blindly; filter with fs.exists first.
5. Retry failed deletions with backoff for transient NameNode issues.

existing = [u for u in urls if fs.exists(u)]
fs.delete(existing)

from apache_beam.io.filesystem import BeamIOError
try:
    fs.delete(urls)
except BeamIOError as e:
    for url, cause in (e.exception_payload or {}).items():
        logger.error('delete failed for %s: %r', url, cause)
    raise

When it happens

Trigger: Calling fs.delete(urls) where at least one URL raises during self._hdfs_client.delete(path, recursive=True): permission denied, NameNode unreachable, or malformed hdfs URL.

Common situations: Cleanup of temp/staging directories owned by another user (permission denied); deleting a batch after resources were already removed by a different job; expired Kerberos tickets; wrong cluster config so paths don't resolve.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• Copy operation failed
• Could not parse url
• Failed to import hdfs. You can ensure it is installed by…
• File not found
• hdfs_host is not set

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/989b38c2f6c0c5be. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/hadoopfilesystem.py:452

    """
    _, path = self._parse_url(url)
    status = self._hdfs_client.status(path, strict=False)
    if status is None:
      raise BeamIOError('File not found: %s' % url)
    return FileMetadata(
        url, status[_FILE_STATUS_LENGTH], status[_FILE_STATUS_UPDATED] / 1000.0)

  def delete(self, urls):
    exceptions = {}
    for url in urls:
      try:
        _, path = self._parse_url(url)
        self._hdfs_client.delete(path, recursive=True)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[url] = e

    if exceptions:
      raise BeamIOError("Delete operation failed", exceptions)

View on GitHub (pinned to 12126d8942)