apache/beam · error · ValueError

Unknown cluster identifier

Error message

Unknown cluster identifier: %s. Cannot create or reusea Dataproc cluster.

What it means

DataprocClusterManager.create() first resolves the given cluster_identifier into ClusterMetadata; if none is found it raises this ValueError and refuses to create or reuse a Dataproc cluster. Note the message template is missing its %s argument, so it prints literally with '%s' — a known bug in Beam.

Solutions

Register the cluster metadata first (e.g. Cache.record_cluster_metadata / ic.record_cluster_metadata_if_allowed) before calling create().
Pass a valid identifier type: a ClusterMetadata, or configure the cluster name/project/region via interactive options.
Print existing known metadata (Cache.list_clusters_metadata) to find a valid identifier.
Inspect self.cluster_metadata(cluster_identifier) to see why resolution returns None (wrong region/project).
If the literal '%s' appears in the message, note the missing identifier is your argument — enable debug logging to confirm.

Example fix

// before
manager = ClustersManager.create(cluster_identifier='some-cluster')
// after
from apache_beam.runners.interactive.dataproc.types import ClusterMetadata
Cache.record_cluster_metadata('session-1', ClusterMetadata(cluster_name='some-cluster', project_id='my-proj', region='us-central1'))
manager = ClustersManager.create(cluster_identifier=ClusterMetadata(cluster_name='some-cluster', project_id='my-proj', region='us-central1'))

Defensive patterns

Strategy: validation

Validate before calling

from apache_beam.runners.interactive.dataproc.cluster_manager import Cache
metadata = Cache.get_cluster_metadata(identifier)
assert metadata is not None, f'Register cluster metadata for {identifier!r} before create()'

Type guard

def identifier_is_known(identifier) -> bool:
    from apache_beam.runners.interactive.dataproc.cluster_manager import ClustersManager
    return ClustersManager.cluster_metadata(identifier) is not None

Try / catch

try:
    manager = ClustersManager.create(cluster_identifier=identifier)
except ValueError as e:
    if 'Unknown cluster identifier' in str(e):
        manager = provision_with_explicit_metadata(identifier)
    else:
        raise

Prevention

Record ClusterMetadata (name/project/region) before calling create()
Pass a ClusterMetadata object instead of a bare string when unsure
Inspect Cache.list_clusters_metadata for known identifiers
Remember the error message prints a literal %s due to a Beam bug; check debug logs for the actual identifier

When it happens

Trigger: Calling ClustersManager.create(cluster_identifier=...) where cluster_identifier is not a known ClouderMetadata (not registered via record/previous detection) and cannot be derived from options, so self.cluster_metadata(cluster_identifier) returns None.

Common situations: Passing a cluster name string without matching project/region/label metadata registered; reusing a manager across notebook restarts where the metadata cache was lost; passing an integer/None identifier; typo in the cluster identifier.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e6fa14dc2184b0b7. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/interactive/interactive_beam.py:433

  # to run.
  # DATAPROC_IMAGE_VERSION = '2.0.XX-debian10'

  def __init__(self) -> None:
    self.dataproc_cluster_managers: dict[ClusterMetadata,
                                         DataprocClusterManager] = {}
    self.master_urls: dict[str, ClusterMetadata] = {}
    self.pipelines: dict[beam.Pipeline, DataprocClusterManager] = {}
    self.default_cluster_metadata: Optional[ClusterMetadata] = None

  def create(
      self, cluster_identifier: ClusterIdentifier) -> DataprocClusterManager:
    """Creates a Dataproc cluster manager provisioned for the cluster
    identified. If the cluster is known, returns an existing cluster manager.
    """
    # Try to get some not-None cluster metadata.
    cluster_metadata = self.cluster_metadata(cluster_identifier)
    if not cluster_metadata:
      raise ValueError(
          'Unknown cluster identifier: %s. Cannot create or reuse'
          'a Dataproc cluster.')
    if not cluster_metadata.region:
      _LOGGER.info(
          'No region information was detected, defaulting Dataproc cluster '
          'region to: us-central1.')
      cluster_metadata.region = 'us-central1'
    elif cluster_metadata.region == 'global':
      # The global region is unsupported as it will be eventually deprecated.
      raise ValueError('Clusters in the global region are not supported.')
    # else use the provided region.
    if (cluster_metadata.num_workers and
        cluster_metadata.num_workers < self.DATAPROC_MINIMUM_WORKER_NUM):
      _LOGGER.info(
          'At least %s workers are required for a cluster, defaulting to %s.',
          self.DATAPROC_MINIMUM_WORKER_NUM,
          self.DATAPROC_MINIMUM_WORKER_NUM)
      cluster_metadata.num_workers = self.DATAPROC_MINIMUM_WORKER_NUM

View on GitHub (pinned to 12126d8942)