{"record":{"id":"10fd0fe94c53bc7d","repo":"apache/beam","slug":"delete-operation-failed","errorCode":null,"errorMessage":"Delete operation failed","messagePattern":"Delete operation failed","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/aws/s3filesystem.py","lineNumber":317,"sourceCode":"      return FileMetadata(\n          path, file_metadata['size'], file_metadata['last_updated'])\n    except Exception as e:  # pylint: disable=broad-except\n      raise BeamIOError(\"Metadata operation failed\", {path: e})\n\n  def delete(self, paths):\n    \"\"\"Deletes files or directories at the provided paths.\n    Directories will be deleted recursively.\n\n    Args:\n      paths: list of paths that give the file objects to be deleted\n    \"\"\"\n    results = s3io.S3IO(options=self._options).delete_paths(paths)\n    exceptions = {\n        path: error\n        for (path, error) in results.items() if error is not None\n    }\n    if exceptions:\n      raise BeamIOError(\"Delete operation failed\", exceptions)\n\n  def report_lineage(self, path, lineage):\n    try:\n      components = s3io.parse_s3_path(path, object_optional=True)\n    except ValueError:\n      # report lineage is fail-safe\n      traceback.print_exc()\n      return\n    if components and not components[-1]:\n      components = components[:-1]\n    lineage.add('s3', *components, last_segment_sep='/')\n","sourceCodeStart":299,"sourceCodeEnd":329,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/aws/s3filesystem.py#L299-L329","documentation":"Raised by S3FileSystem.delete when one or more of the requested paths failed to be deleted on S3. BeamIOError carries a {path: error} map of the failures; paths that deleted successfully are not reported.","triggerScenarios":"Calling fs.delete(paths) where any path is missing, is a non-empty prefix lacking ListBucket/DeleteObject permission, or S3 returns errors (throttling, AccessDenied).","commonSituations":"Cleanup jobs racing with writers re-creating objects; IAM policy missing s3:DeleteObject or s3:DeleteObjectVersion; deleting versioned objects without version permissions.","solutions":["Inspect BeamIOError.exception_details to see which paths failed and why","Ensure IAM permissions include s3:DeleteObject (and version variants for versioned buckets)","Pre-check paths with exists() and tolerate already-deleted paths","Retry the failed subset after fixing permissions/rate limits"],"exampleFix":"// before\nfs.delete(stale_paths)\n// after\nfrom apache_beam.io.filesystem import BeamIOError\ntry:\n    fs.delete(stale_paths)\nexcept BeamIOError as e:\n    failed = list((e.exception_details or {}).keys())\n    logging.error('failed deletions: %s', failed)","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    fs.delete(paths)\nexcept BeamIOError as e:\n    failed = (e.exception_details or {}).keys()\n    schedule_retry(list(failed))","preventionTips":["Grant s3:DeleteObject (and versioned variants) to the job role","Treat already-deleted paths as success in cleanup logic","Retry throttled deletions with exponential backoff"],"tags":["s3","io","aws","delete","permissions"],"backgroundTag":"permission-denied","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}