apache/beam · error · ValueError

ib.options.cache_root needs to be a Cloud Storage Bucket to…

Error message

ib.options.cache_root needs to be a Cloud Storage Bucket to cache source recording and PCollections in current interactive setup, instead '{cache_dir}' is assigned.

What it means

ClusterManager for Dataproc requires the interactive options cache_root to be a Cloud Storage bucket path (gs://...) because staged init actions and recorded caches must live in GCS. A non-GCS cache_dir is logged and raises ValueError.

Solutions

Set ib.options.cache_root = 'gs://<your-bucket>/<path>' before creating the cluster manager.
Create/verify the GCS bucket exists and your credentials can write to it.
Fix the URL scheme typos (must start exactly with 'gs://').

Example fix

// before
ib.options.cache_root = '/tmp/ib_cache'
// after
ib.options.cache_root = 'gs://my-bucket/ib-cache'

Defensive patterns

Strategy: validation

Validate before calling

import ib
root = ib.options.cache_root
if using_dataproc_cluster_manager and (not root or not root.startswith('gs://')):
    raise ValueError('cache_root must be a gs:// bucket for Dataproc interactive runs.')

Try / catch

try:
    manager = ClusterManager(cache_dir=ib.options.cache_root, ...)
except ValueError as e:
    if 'gs://' in str(e):
        ib.options.cache_root = 'gs://my-bucket/ib-cache'
        manager = ClusterManager(cache_dir=ib.options.cache_root, ...)

Prevention

Always configure ib.options.cache_root = 'gs://bucket/path' before instantiating Dataproc cluster managers.
Verify bucket existence and write access before the run.

When it happens

Trigger: Constructing ClusterManager (e.g. FlinkOnDataproc) with a cache_dir such as a local path or 's3://...' that doesn't start with 'gs://'; ib.options.cache_root left as a local temp dir while a Dataproc cluster manager is used.

Common situations: Running interactive Beam against Dataproc without setting ib.options.cache_root to a GCS bucket; typo'd GCS URLs ('gcs://', missing 'gs://'); reusing local cache config from a laptop run.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/479a861cf2d09d2e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/interactive/dataproc/dataproc_cluster_manager.py:103

    """
    self.cluster_metadata = cluster_metadata
    # Pipelines whose jobs are executed on the cluster.
    self.pipelines = set()
    self._cluster_client = dataproc_v1.ClusterControllerClient(
        client_options={
            'api_endpoint': \
            f'{self.cluster_metadata.region}-dataproc.googleapis.com:443'
        })
    self._fs = gcsfilesystem.GCSFileSystem(PipelineOptions())
    self._staging_directory = None
    cache_dir = ie.current_env().options.cache_root
    if not cache_dir.startswith('gs://'):
      error_msg = (
          'ib.options.cache_root needs to be a Cloud Storage '
          'Bucket to cache source recording and PCollections in current '
          f'interactive setup, instead \'{cache_dir}\' is assigned.')
      _LOGGER.error(error_msg)
      raise ValueError(error_msg)
    self._cache_root = cache_dir.rstrip('/')

  def stage_init_action(self) -> str:
    """Stages the initialization action script to GCS cache root to set up
    Dataproc clusters.

    Returns the staged gcs file path.
    """
    # Versionizes the initialization action script.
    init_action_ver = obfuscate(INIT_ACTION)
    path = f'{self._cache_root}/dataproc-init-action-{init_action_ver}.sh'
    if not self._fs.exists(path):
      with self._fs.create(path) as bwriter:
        bwriter.write(INIT_ACTION.encode())
    return path

  @progress_indicated
  def create_cluster(self, cluster: dict) -> None:

View on GitHub (pinned to 12126d8942)