apache/beam · error · ValueError

Clusters in the global region are not supported.

Error message

Clusters in the global region are not supported.

What it means

DataprocClusterManager.create() rejects clusters whose region is 'global', raising this ValueError. Google is deprecating the Dataproc global region, so Beam's interactive runner explicitly refuses to provision clusters there instead of defaulting to a concrete region.

Solutions

  1. Set an explicit region (e.g. 'us-central1') in ClusterMetadata instead of 'global'.
  2. Configure the region via the interactive environment option (cluster region) or GOOGLE_CLOUD_DATAPROC_REGION appropriately.
  3. Migrate any existing 'global' Dataproc clusters to a regional endpoint; recreate the cluster regionally.
  4. Update gcloud config: gcloud config set dataproc/region us-central1 so defaults stop yielding 'global'.

Example fix

// before
metadata = ClusterMetadata(cluster_name='c1', project_id='proj', region='global')
manager = ClustersManager.create(cluster_identifier=metadata)
// after
metadata = ClusterMetadata(cluster_name='c1', project_id='proj', region='us-central1')
manager = ClustersManager.create(cluster_identifier=metadata)
Defensive patterns

Strategy: validation

Validate before calling

metadata = ClusterMetadata(cluster_name=name, project_id=project, region=region)
assert metadata.region != 'global', 'Dataproc global region is unsupported; pick a regional endpoint'

Type guard

def region_is_supported(region: str) -> bool:
    return bool(region) and region != 'global'

Try / catch

try:
    manager = ClustersManager.create(cluster_identifier=metadata)
except ValueError as e:
    if 'global region' in str(e):
        metadata.region = 'us-central1'
        manager = ClustersManager.create(cluster_identifier=metadata)
    else:
        raise

Prevention

When it happens

Trigger: Calling create() with ClusterMetadata whose region == 'global' (or the interactive option/DPZ config resolving to 'global'), e.g. metadata derived from an old configuration or an environment where region defaults to 'global'.

Common situations: Legacy Dataproc configs that used region 'global'; gcloud default region unset so code paths fill 'global'; migrating older notebooks to newer Beam where regional clusters are mandatory.

Understand the failure class

Background: "is deprecated and will be removed" — deprecation warnings for old API names, keywords, and options, and how to migrate before the removal release — this error's family across 29 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4a63624dd86d7aa9. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/interactive/interactive_beam.py:443

  def create(
      self, cluster_identifier: ClusterIdentifier) -> DataprocClusterManager:
    """Creates a Dataproc cluster manager provisioned for the cluster
    identified. If the cluster is known, returns an existing cluster manager.
    """
    # Try to get some not-None cluster metadata.
    cluster_metadata = self.cluster_metadata(cluster_identifier)
    if not cluster_metadata:
      raise ValueError(
          'Unknown cluster identifier: %s. Cannot create or reuse'
          'a Dataproc cluster.')
    if not cluster_metadata.region:
      _LOGGER.info(
          'No region information was detected, defaulting Dataproc cluster '
          'region to: us-central1.')
      cluster_metadata.region = 'us-central1'
    elif cluster_metadata.region == 'global':
      # The global region is unsupported as it will be eventually deprecated.
      raise ValueError('Clusters in the global region are not supported.')
    # else use the provided region.
    if (cluster_metadata.num_workers and
        cluster_metadata.num_workers < self.DATAPROC_MINIMUM_WORKER_NUM):
      _LOGGER.info(
          'At least %s workers are required for a cluster, defaulting to %s.',
          self.DATAPROC_MINIMUM_WORKER_NUM,
          self.DATAPROC_MINIMUM_WORKER_NUM)
      cluster_metadata.num_workers = self.DATAPROC_MINIMUM_WORKER_NUM
    known_dcm = self.dataproc_cluster_managers.get(cluster_metadata, None)
    if known_dcm:
      return known_dcm
    dcm = DataprocClusterManager(cluster_metadata)
    dcm.create_flink_cluster()
    # ClusterMetadata with derivative fields populated by the dcm.
    derived_meta = dcm.cluster_metadata
    self.dataproc_cluster_managers[derived_meta] = dcm
    self.master_urls[derived_meta.master_url] = derived_meta
    # Update the default cluster metadata to the one just created.

View on GitHub (pinned to 12126d8942)