apache/beam · error · ValueError

You cannot create a cluster in project

Error message

You cannot create a cluster in project: {}

What it means

When the Dataproc API returns HTTP 403 during create_cluster, the manager translates it into ValueError('You cannot create a cluster in project: ...'), indicating the credentials lack permission to create clusters in that project.

Solutions

1. Grant the caller the Dataproc Admin/Editor role (dataproc.clusters.create) on the project.
2. Verify project_id in ClusterMetadata is the intended, correct project.
3. Check that the Dataproc service agent has iam.serviceAccountUser permission on the compute service account.
4. Confirm credentials (gcloud auth application-default login / service account key) belong to an account with access.

Example fix

// before
# caller lacks roles; fails with 403
cluster = manager.create_flink_cluster()
// after
# grant: gcloud projects add-iam-policy-binding PROJECT --member=user:me@example.com --role=roles/dataproc.editor
cluster = manager.create_flink_cluster()

# preflight IAM check
from google.cloud import dataproc_v1
client = dataproc_v1.ClusterControllerClient(client_options={'api_endpoint': f'{region}-dataproc.googleapis.com:443'})
# caller must have dataproc.clusters.create; test with a dry list call
try:
    client.list_clusters(request={'project_id': project, 'region': region})
except Exception as e:
    check_permission(e)

try:
    cluster = manager.create_flink_cluster()
except ValueError as e:
    if 'You cannot create a cluster in project' in str(e):
        grant_dataproc_roles() or switch project/service account
    else:
        raise

When it happens

Trigger: Calling create_cluster (via create_flink_cluster) where the service account/user lacks dataproc.clusters.create on the project; 403 from the Dataproc regions.create API call.

Common situations: Using default credentials without Dataproc roles; wrong project id; org policy or service account restrictions; missing iam.serviceAccountUser role for the Dataproc service agent.

Understand the failure class

Background: "You do not have permission" / 403 Forbidden errors: authenticated but not allowed — causes and fixes across open-source libraries — this error's family across 31 libraries.

Related errors

• You cannot delete a cluster in project
• Region does not exist!
• You cannot view clusters in project
• A cluster_identifier should be Optional[Union[str…
• A VPC network must be provided to use a private endpoint.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9ac344cc52567dbd. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/interactive/dataproc/dataproc_cluster_manager.py:149

      return
    try:
      self._cluster_client.create_cluster(
          request={
              'project_id': self.cluster_metadata.project_id,
              'region': self.cluster_metadata.region,
              'cluster': cluster
          })
    except Exception as e:
      if e.code == 409:
        _LOGGER.info(
            'Cluster %s already exists. Continuing...',
            self.cluster_metadata.cluster_name)
      elif e.code == 403:
        _LOGGER.error(
            'Due to insufficient project permissions, '
            'unable to create cluster: %s',
            self.cluster_metadata.cluster_name)
        raise ValueError(
            'You cannot create a cluster in project: {}'.format(
                self.cluster_metadata.project_id))
      elif e.code == 501:
        _LOGGER.error(
            'Invalid region provided: %s', self.cluster_metadata.region)
        raise ValueError(
            'Region {} does not exist!'.format(self.cluster_metadata.region))
      else:
        _LOGGER.error(
            'Unable to create cluster: %s', self.cluster_metadata.cluster_name)
        raise e
    else:
      _LOGGER.info(
          'Cluster created successfully: %s',
          self.cluster_metadata.cluster_name)
      self._staging_directory = self.get_staging_location()
      master_url, dashboard = self.get_master_url_and_dashboard()
      self.cluster_metadata.master_url = master_url

View on GitHub (pinned to 12126d8942)