{"record":{"id":"8632504c21de97af","repo":"apache/beam","slug":"cluster-was-not-found","errorCode":null,"errorMessage":"Cluster was not found: {}","messagePattern":"Cluster was not found: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":404,"severity":"error","filePath":"sdks/python/apache_beam/runners/interactive/dataproc/dataproc_cluster_manager.py","lineNumber":256,"sourceCode":"          request={\n              'project_id': self.cluster_metadata.project_id,\n              'region': self.cluster_metadata.region,\n              'cluster_name': self.cluster_metadata.cluster_name,\n          })\n      self.cleanup_staging_files()\n    except Exception as e:\n      if e.code == 403:\n        _LOGGER.error(\n            'Due to insufficient project permissions, '\n            'unable to clean up the default cluster: %s',\n            self.cluster_metadata.cluster_name)\n        raise ValueError(\n            'You cannot delete a cluster in project: {}'.format(\n                self.cluster_metadata.project_id))\n      elif e.code == 404:\n        _LOGGER.error(\n            'Cluster does not exist: %s', self.cluster_metadata.cluster_name)\n        raise ValueError(\n            'Cluster was not found: {}'.format(\n                self.cluster_metadata.cluster_name))\n      else:\n        _LOGGER.error(\n            'Failed to delete cluster: %s', self.cluster_metadata.cluster_name)\n        raise e\n\n  def get_cluster_details(self) -> dataproc_v1.Cluster:\n    \"\"\"Gets the Dataproc_v1 Cluster object for the current cluster manager.\"\"\"\n    try:\n      return self._cluster_client.get_cluster(\n          request={\n              'project_id': self.cluster_metadata.project_id,\n              'region': self.cluster_metadata.region,\n              'cluster_name': self.cluster_metadata.cluster_name\n          })\n    except Exception as e:\n      if e.code == 403:","sourceCodeStart":238,"sourceCodeEnd":274,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/interactive/dataproc/dataproc_cluster_manager.py#L238-L274","documentation":"Raised by ClustersManager.cleanup() when the Dataproc API returns HTTP 404 while attempting to delete a cluster. The Apache Beam interactive runner wraps that failure in a ValueError because the cluster it was asked to delete no longer exists in the given project. It is typically raised during teardown of an interactive Dataproc session after the cluster was already removed.","triggerScenarios":"Calling cleanup() (directly or via _cleanup) when self.cluster_metadata.cluster_name does not resolve to an existing cluster in the project — e.g. the cluster was deleted by another process, expired (max-age TTL), or was created in a different region than queried.","commonSituations":"Double teardown in a notebook (cleanup called twice); cluster auto-deleted by a Dataproc TTL or idle-deletion policy; deleting a cluster from a second notebook session; cloud project/region mismatch causing a 404.","solutions":["Check whether the cluster still exists (Dataproc console or gcloud dataproc clusters list) before/around calling cleanup().","Wrap cleanup() in try/except ValueError and treat 404-style 'Cluster was not found' as an already-clean state, not a failure.","Avoid running cleanup twice on the same cluster manager; track teardown state in the notebook.","Verify project_id, region and cluster_name in the ClusterMetadata match the actually provisioned cluster.","If the cluster vanished unexpectedly, check Dataproc TTL/idle-deletion settings and audit logs for who deleted it."],"exampleFix":"// before\nclusters_manager.cleanup()\n// after\ntry:\n    clusters_manager.cleanup()\nexcept ValueError as e:\n    if 'Cluster was not found' in str(e):\n        _LOGGER.info('Cluster already deleted; nothing to clean up.')\n    else:\n        raise","handlingStrategy":"try-catch","validationCode":"exists = subprocess.run(['gcloud', 'dataproc', 'clusters', 'describe', name, '--region', region, '--project', project], capture_output=True).returncode == 0\nif not exists:\n    skip_cleanup = True","typeGuard":null,"tryCatchPattern":"try:\n    clusters_manager.cleanup()\nexcept ValueError as e:\n    if 'Cluster was not found' in str(e):\n        pass  # already deleted\n    else:\n        raise","preventionTips":["Never call cleanup() twice on the same manager","Track cluster teardown state in the notebook","Check Dataproc TTL/idle-deletion policies on the project","Confirm cluster still exists before programmatic deletion"],"tags":["gcp","dataproc","resource-not-found","python"],"backgroundTag":"resource-not-found","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}