apache/beam · error · ValueError

You cannot delete a cluster in project

Error message

You cannot delete a cluster in project: {}

What it means

During cluster cleanup/delete, a Dataproc 403 (insufficient permissions) is re-raised as ValueError('You cannot delete a cluster in project: ...'), meaning the caller cannot delete clusters in that project.

Solutions

1. Grant roles/dataproc.editor or dataproc.clusters.delete permission to the caller in that project.
2. Delete the cluster with an account/service account that created it or has admin rights.
3. Verify project_id is correct; you may be deleting in a different project than intended.
4. As fallback, manually remove the orphaned cluster via console/gcloud with sufficient credentials.

Example fix

// before
# 403 on delete
manager.cleanup()
// after
# grant: gcloud projects add-iam-policy-binding PROJECT --member=... --role=roles/dataproc.editor
manager.cleanup()

# preflight permission: attempt a harmless dataproc call
from google.cloud import dataproc_v1
client = dataproc_v1.ClusterControllerClient(client_options={'api_endpoint': f'{region}-dataproc.googleapis.com:443'})
clusters = client.list_clusters(request={'project_id': project, 'region': region})  # raises early if no access

try:
    manager.cleanup()
except ValueError as e:
    if 'You cannot delete a cluster in project' in str(e):
        escalate_credentials() or delete_manually_with_admin()  # avoid orphaned clusters
    else:
        raise

When it happens

Trigger: cleanup()/_cleanup invoked (e.g. notebook teardown or explicit delete) where the Dataproc delete API returns 403 due to missing dataproc.clusters.delete permission on the project.

Common situations: Notebook user's credentials differ from the creator's; viewer-only Dataproc role; service account restrictions or org policy preventing deletion.

Understand the failure class

Background: "You do not have permission" / 403 Forbidden errors: authenticated but not allowed — causes and fixes across open-source libraries — this error's family across 31 libraries.

Related errors

• You cannot create a cluster in project
• Region does not exist!
• You cannot view clusters in project
• A cluster_identifier should be Optional[Union[str…
• A VPC network must be provided to use a private endpoint.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/17525aacc15f53ff. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/interactive/dataproc/dataproc_cluster_manager.py:250

  def cleanup(self) -> None:
    """Deletes the cluster that uses the attributes initialized
    with the DataprocClusterManager instance."""
    try:
      self._cluster_client.delete_cluster(
          request={
              'project_id': self.cluster_metadata.project_id,
              'region': self.cluster_metadata.region,
              'cluster_name': self.cluster_metadata.cluster_name,
          })
      self.cleanup_staging_files()
    except Exception as e:
      if e.code == 403:
        _LOGGER.error(
            'Due to insufficient project permissions, '
            'unable to clean up the default cluster: %s',
            self.cluster_metadata.cluster_name)
        raise ValueError(
            'You cannot delete a cluster in project: {}'.format(
                self.cluster_metadata.project_id))
      elif e.code == 404:
        _LOGGER.error(
            'Cluster does not exist: %s', self.cluster_metadata.cluster_name)
        raise ValueError(
            'Cluster was not found: {}'.format(
                self.cluster_metadata.cluster_name))
      else:
        _LOGGER.error(
            'Failed to delete cluster: %s', self.cluster_metadata.cluster_name)
        raise e

  def get_cluster_details(self) -> dataproc_v1.Cluster:
    """Gets the Dataproc_v1 Cluster object for the current cluster manager."""
    try:
      return self._cluster_client.get_cluster(
          request={

View on GitHub (pinned to 12126d8942)