apache/beam · error · ValueError

You cannot view clusters in project

Error message

You cannot view clusters in project: {}

What it means

Raised by ClustersManager.get_cluster_details() when the Dataproc API returns HTTP 403 for a cluster info request. Beam wraps this in a ValueError indicating the credentials in use lack permission to view clusters in the given project. Callers include wait_for_cluster_to_provision, get_staging_location and parse_master_url_and_dashboard, so provisioning can also fail with this.

Solutions

1. Grant the account/dataproc service account the roles/dataproc.editor (or at least roles/dataproc.viewer) IAM role on the project.
2. Verify the correct project_id in cluster_metadata; you may be querying the wrong project.
3. Re-authenticate with sufficient credentials (gcloud auth application-default login) and restart the notebook kernel.
4. Ensure the Dataproc API (dataproc.googleapis.com) is enabled in the target project.
5. Check organization policies / VPC-SC perimeter that may deny Dataproc reads.

Example fix

// before (shell)
gcloud auth application-default login  # account without dataproc access
// after
gcloud auth application-default login  # account with roles/dataproc.viewer
gcloud projects add-iam-policy-binding PROJECT_ID \
  --member='serviceAccount:SA@PROJECT.iam.gserviceaccount.com' \
  --role='roles/dataproc.viewer'

perm = subprocess.run(['gcloud', 'projects', 'get-iam-policy', project, '--flatten=bindings', '--filter=bindings.members:' + member, '--format=value(bindings.role)'], capture_output=True, text=True).stdout
assert 'dataproc' in perm, 'Missing Dataproc IAM role'

try:
    clusters_manager.get_cluster_details()
except ValueError as e:
    if 'cannot view clusters' in str(e):
        raise PermissionError('Grant roles/dataproc.viewer on the project') from e
    raise

When it happens

Trigger: Calling get_cluster_details() (or any caller: wait_for_cluster_to_provision, get_staging_location, parse_master_url_and_dashboard) when the authenticated account lacks dataproc.clusters.get on the project, or the Dataproc API is not enabled / service account lacks the Dataproc Viewer role.

Common situations: Using a default service account without Dataproc roles; running in a notebook on a machine with restricted ADC credentials; organization policy blocking access; wrong project_id in ClusterMetadata pointing at a project the user cannot see.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• Cluster was not found
• Clusters in the global region are not supported.
• Unknown cluster identifier
• You cannot create a cluster in project
• You cannot delete a cluster in project

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/78be16d77ff4d315. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/interactive/dataproc/dataproc_cluster_manager.py:279

            'Failed to delete cluster: %s', self.cluster_metadata.cluster_name)
        raise e

  def get_cluster_details(self) -> dataproc_v1.Cluster:
    """Gets the Dataproc_v1 Cluster object for the current cluster manager."""
    try:
      return self._cluster_client.get_cluster(
          request={
              'project_id': self.cluster_metadata.project_id,
              'region': self.cluster_metadata.region,
              'cluster_name': self.cluster_metadata.cluster_name
          })
    except Exception as e:
      if e.code == 403:
        _LOGGER.error(
            'Due to insufficient project permissions, '
            'unable to retrieve information for cluster: %s',
            self.cluster_metadata.cluster_name)
        raise ValueError(
            'You cannot view clusters in project: {}'.format(
                self.cluster_metadata.project_id))
      elif e.code == 404:
        _LOGGER.error(
            'Cluster does not exist: %s', self.cluster_metadata.cluster_name)
        raise ValueError(
            'Cluster was not found: {}'.format(
                self.cluster_metadata.cluster_name))
      else:
        _LOGGER.error(
            'Failed to get information for cluster: %s',
            self.cluster_metadata.cluster_name)
        raise e

  def wait_for_cluster_to_provision(self) -> None:
    while self.get_cluster_details().status.state.name == 'CREATING':
      time.sleep(15)

View on GitHub (pinned to 12126d8942)