apache/beam · error · ValueError

Region does not exist!

Error message

Region {} does not exist!

What it means

A Dataproc API 501 (or 'region not found') response during create_cluster is converted to ValueError('Region {} does not exist!'), meaning the region string in ClusterMetadata is not a valid Dataproc region.

Solutions

1. Set a valid Dataproc region in ClusterMetadata (e.g. 'us-central1', 'europe-west1').
2. Run `gcloud dataproc regions list --project PROJECT` to see valid regions.
3. Ensure the region is one where the Dataproc API is available/enabled.
4. Fix casing and format: lowercase region, no zone suffix.

Example fix

// before
ClusterMetadata(project_id='p', region='us-central1-a', ...)
// after
ClusterMetadata(project_id='p', region='us-central1', ...)

import subprocess
regions = subprocess.run(['gcloud', 'dataproc', 'regions', 'list', '--project', project,
                          '--format=value(name)'], capture_output=True, text=True).stdout.split()
assert region in regions, f'{region} is not a valid Dataproc region'

try:
    cluster = manager.create_flink_cluster()
except ValueError as e:
    if 'does not exist' in str(e) and 'Region' in str(e):
        manager.cluster_metadata.region = 'us-central1'
        cluster = manager.create_flink_cluster()

When it happens

Trigger: create_cluster called with an invalid region name (typo, unsupported region, or region where the Dataproc API isn't enabled) — the API responds 501/NOT_IMPLEMENTED and the manager raises this error.

Common situations: Passing a zone instead of a region ('us-central1-a' vs 'us-central1'); using a newly announced region not yet supported; wrong region casing; missing '-a' style suffixes misconceptions.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• You cannot create a cluster in project
• You cannot delete a cluster in project
• A cluster_identifier should be Optional[Union[str…
• A VPC network must be provided to use a private endpoint.
• Both deidentification_template_name and…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f3c757853d8394b1. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/interactive/dataproc/dataproc_cluster_manager.py:155

              'cluster': cluster
          })
    except Exception as e:
      if e.code == 409:
        _LOGGER.info(
            'Cluster %s already exists. Continuing...',
            self.cluster_metadata.cluster_name)
      elif e.code == 403:
        _LOGGER.error(
            'Due to insufficient project permissions, '
            'unable to create cluster: %s',
            self.cluster_metadata.cluster_name)
        raise ValueError(
            'You cannot create a cluster in project: {}'.format(
                self.cluster_metadata.project_id))
      elif e.code == 501:
        _LOGGER.error(
            'Invalid region provided: %s', self.cluster_metadata.region)
        raise ValueError(
            'Region {} does not exist!'.format(self.cluster_metadata.region))
      else:
        _LOGGER.error(
            'Unable to create cluster: %s', self.cluster_metadata.cluster_name)
        raise e
    else:
      _LOGGER.info(
          'Cluster created successfully: %s',
          self.cluster_metadata.cluster_name)
      self._staging_directory = self.get_staging_location()
      master_url, dashboard = self.get_master_url_and_dashboard()
      self.cluster_metadata.master_url = master_url
      self.cluster_metadata.dashboard = dashboard

  def create_flink_cluster(self) -> None:
    """Calls _create_cluster with a configuration that enables FlinkRunner."""
    init_action_path = self.stage_init_action()
    # https://cloud.google.com/php/docs/reference/cloud-dataproc/latest/V1.Cluster

View on GitHub (pinned to 12126d8942)