{"record":{"id":"479a861cf2d09d2e","repo":"apache/beam","slug":"ib-options-cache-root-needs-to-be-a-cloud-storage-bucket-to","errorCode":null,"errorMessage":"ib.options.cache_root needs to be a Cloud Storage Bucket to cache source recording and PCollections in current interactive setup, instead '{cache_dir}' is assigned.","messagePattern":"ib\\.options\\.cache_root needs to be a Cloud Storage Bucket to cache source recording and PCollections in current interactive setup, instead '(.+?)' is assigned\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/runners/interactive/dataproc/dataproc_cluster_manager.py","lineNumber":103,"sourceCode":"    \"\"\"\n    self.cluster_metadata = cluster_metadata\n    # Pipelines whose jobs are executed on the cluster.\n    self.pipelines = set()\n    self._cluster_client = dataproc_v1.ClusterControllerClient(\n        client_options={\n            'api_endpoint': \\\n            f'{self.cluster_metadata.region}-dataproc.googleapis.com:443'\n        })\n    self._fs = gcsfilesystem.GCSFileSystem(PipelineOptions())\n    self._staging_directory = None\n    cache_dir = ie.current_env().options.cache_root\n    if not cache_dir.startswith('gs://'):\n      error_msg = (\n          'ib.options.cache_root needs to be a Cloud Storage '\n          'Bucket to cache source recording and PCollections in current '\n          f'interactive setup, instead \\'{cache_dir}\\' is assigned.')\n      _LOGGER.error(error_msg)\n      raise ValueError(error_msg)\n    self._cache_root = cache_dir.rstrip('/')\n\n  def stage_init_action(self) -> str:\n    \"\"\"Stages the initialization action script to GCS cache root to set up\n    Dataproc clusters.\n\n    Returns the staged gcs file path.\n    \"\"\"\n    # Versionizes the initialization action script.\n    init_action_ver = obfuscate(INIT_ACTION)\n    path = f'{self._cache_root}/dataproc-init-action-{init_action_ver}.sh'\n    if not self._fs.exists(path):\n      with self._fs.create(path) as bwriter:\n        bwriter.write(INIT_ACTION.encode())\n    return path\n\n  @progress_indicated\n  def create_cluster(self, cluster: dict) -> None:","sourceCodeStart":85,"sourceCodeEnd":121,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/interactive/dataproc/dataproc_cluster_manager.py#L85-L121","documentation":"ClusterManager for Dataproc requires the interactive options cache_root to be a Cloud Storage bucket path (gs://...) because staged init actions and recorded caches must live in GCS. A non-GCS cache_dir is logged and raises ValueError.","triggerScenarios":"Constructing ClusterManager (e.g. FlinkOnDataproc) with a cache_dir such as a local path or 's3://...' that doesn't start with 'gs://'; ib.options.cache_root left as a local temp dir while a Dataproc cluster manager is used.","commonSituations":"Running interactive Beam against Dataproc without setting ib.options.cache_root to a GCS bucket; typo'd GCS URLs ('gcs://', missing 'gs://'); reusing local cache config from a laptop run.","solutions":["Set ib.options.cache_root = 'gs://<your-bucket>/<path>' before creating the cluster manager.","Create/verify the GCS bucket exists and your credentials can write to it.","Fix the URL scheme typos (must start exactly with 'gs://')."],"exampleFix":"// before\nib.options.cache_root = '/tmp/ib_cache'\n// after\nib.options.cache_root = 'gs://my-bucket/ib-cache'","handlingStrategy":"validation","validationCode":"import ib\nroot = ib.options.cache_root\nif using_dataproc_cluster_manager and (not root or not root.startswith('gs://')):\n    raise ValueError('cache_root must be a gs:// bucket for Dataproc interactive runs.')","typeGuard":null,"tryCatchPattern":"try:\n    manager = ClusterManager(cache_dir=ib.options.cache_root, ...)\nexcept ValueError as e:\n    if 'gs://' in str(e):\n        ib.options.cache_root = 'gs://my-bucket/ib-cache'\n        manager = ClusterManager(cache_dir=ib.options.cache_root, ...)","preventionTips":["Always configure ib.options.cache_root = 'gs://bucket/path' before instantiating Dataproc cluster managers.","Verify bucket existence and write access before the run."],"tags":["apache-beam","interactive-beam","dataproc","gcs","config"],"backgroundTag":"invalid-config-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}