{"record":{"id":"989b38c2f6c0c5be","repo":"apache/beam","slug":"delete-operation-failed-hadoopfilesystem","errorCode":null,"errorMessage":"Delete operation failed","messagePattern":"Delete operation failed","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/hadoopfilesystem.py","lineNumber":452,"sourceCode":"    \"\"\"\n    _, path = self._parse_url(url)\n    status = self._hdfs_client.status(path, strict=False)\n    if status is None:\n      raise BeamIOError('File not found: %s' % url)\n    return FileMetadata(\n        url, status[_FILE_STATUS_LENGTH], status[_FILE_STATUS_UPDATED] / 1000.0)\n\n  def delete(self, urls):\n    exceptions = {}\n    for url in urls:\n      try:\n        _, path = self._parse_url(url)\n        self._hdfs_client.delete(path, recursive=True)\n      except Exception as e:  # pylint: disable=broad-except\n        exceptions[url] = e\n\n    if exceptions:\n      raise BeamIOError(\"Delete operation failed\", exceptions)\n","sourceCodeStart":434,"sourceCodeEnd":453,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/hadoopfilesystem.py#L434-L453","documentation":"HadoopFileSystem.delete attempts to delete each URL recursively, collecting any exception per URL; if any deletion fails it raises a single BeamIOError 'Delete operation failed' with the per-URL causes in exception_payload. Permission errors, connection errors, or URL parse failures land here.","triggerScenarios":"Calling fs.delete(urls) where at least one URL raises during self._hdfs_client.delete(path, recursive=True): permission denied, NameNode unreachable, or malformed hdfs URL.","commonSituations":"Cleanup of temp/staging directories owned by another user (permission denied); deleting a batch after resources were already removed by a different job; expired Kerberos tickets; wrong cluster config so paths don't resolve.","solutions":["Inspect BeamIOError.exception_payload to identify which URLs failed and the underlying error.","Check HDFS permissions on the paths and the deleting user (hdfs dfs -chmod/-chown, or run as the file owner).","Verify Kerberos auth is valid (kinit) and HADOOP_CONF_DIR points to the correct cluster.","Skip URLs that no longer exist instead of deleting them blindly; filter with fs.exists first.","Retry failed deletions with backoff for transient NameNode issues."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"existing = [u for u in urls if fs.exists(u)]\nfs.delete(existing)","typeGuard":null,"tryCatchPattern":"from apache_beam.io.filesystem import BeamIOError\ntry:\n    fs.delete(urls)\nexcept BeamIOError as e:\n    for url, cause in (e.exception_payload or {}).items():\n        logger.error('delete failed for %s: %r', url, cause)\n    raise","preventionTips":["Only delete paths your user owns or has write permission on.","Renew Kerberos tickets before long cleanup jobs.","Filter out non-existent URLs before deleting.","Retry per-URL failures using exception_payload."],"tags":["python","apache-beam","hdfs","delete"],"backgroundTag":"permission-denied","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}