apache/beam · error · ValueError

Cluster was not found

Error message

Cluster was not found: {}

What it means

Raised by ClustersManager.cleanup() when the Dataproc API returns HTTP 404 while attempting to delete a cluster. The Apache Beam interactive runner wraps that failure in a ValueError because the cluster it was asked to delete no longer exists in the given project. It is typically raised during teardown of an interactive Dataproc session after the cluster was already removed.

Solutions

1. Check whether the cluster still exists (Dataproc console or gcloud dataproc clusters list) before/around calling cleanup().
2. Wrap cleanup() in try/except ValueError and treat 404-style 'Cluster was not found' as an already-clean state, not a failure.
3. Avoid running cleanup twice on the same cluster manager; track teardown state in the notebook.
4. Verify project_id, region and cluster_name in the ClusterMetadata match the actually provisioned cluster.
5. If the cluster vanished unexpectedly, check Dataproc TTL/idle-deletion settings and audit logs for who deleted it.

Example fix

// before
clusters_manager.cleanup()
// after
try:
    clusters_manager.cleanup()
except ValueError as e:
    if 'Cluster was not found' in str(e):
        _LOGGER.info('Cluster already deleted; nothing to clean up.')
    else:
        raise

exists = subprocess.run(['gcloud', 'dataproc', 'clusters', 'describe', name, '--region', region, '--project', project], capture_output=True).returncode == 0
if not exists:
    skip_cleanup = True

try:
    clusters_manager.cleanup()
except ValueError as e:
    if 'Cluster was not found' in str(e):
        pass  # already deleted
    else:
        raise

When it happens

Trigger: Calling cleanup() (directly or via _cleanup) when self.cluster_metadata.cluster_name does not resolve to an existing cluster in the project — e.g. the cluster was deleted by another process, expired (max-age TTL), or was created in a different region than queried.

Common situations: Double teardown in a notebook (cleanup called twice); cluster auto-deleted by a Dataproc TTL or idle-deletion policy; deleting a cluster from a second notebook session; cloud project/region mismatch causing a 404.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Clusters in the global region are not supported.
• Dataset does not exist in your project. You have to create…
• Unknown cluster identifier
• You cannot view clusters in project
• A cluster_identifier should be Optional[Union[str…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8632504c21de97af. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/interactive/dataproc/dataproc_cluster_manager.py:256

          request={
              'project_id': self.cluster_metadata.project_id,
              'region': self.cluster_metadata.region,
              'cluster_name': self.cluster_metadata.cluster_name,
          })
      self.cleanup_staging_files()
    except Exception as e:
      if e.code == 403:
        _LOGGER.error(
            'Due to insufficient project permissions, '
            'unable to clean up the default cluster: %s',
            self.cluster_metadata.cluster_name)
        raise ValueError(
            'You cannot delete a cluster in project: {}'.format(
                self.cluster_metadata.project_id))
      elif e.code == 404:
        _LOGGER.error(
            'Cluster does not exist: %s', self.cluster_metadata.cluster_name)
        raise ValueError(
            'Cluster was not found: {}'.format(
                self.cluster_metadata.cluster_name))
      else:
        _LOGGER.error(
            'Failed to delete cluster: %s', self.cluster_metadata.cluster_name)
        raise e

  def get_cluster_details(self) -> dataproc_v1.Cluster:
    """Gets the Dataproc_v1 Cluster object for the current cluster manager."""
    try:
      return self._cluster_client.get_cluster(
          request={
              'project_id': self.cluster_metadata.project_id,
              'region': self.cluster_metadata.region,
              'cluster_name': self.cluster_metadata.cluster_name
          })
    except Exception as e:
      if e.code == 403:

View on GitHub (pinned to 12126d8942)