{"record":{"id":"f8ddd94aeec3bcde","repo":"apache/beam","slug":"gcs-cache-paths-are-not-currently-supported-for-streaming","errorCode":null,"errorMessage":"GCS cache paths are not currently supported for streaming pipelines.","messagePattern":"GCS cache paths are not currently supported for streaming pipelines\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/runners/interactive/background_caching_job.py","lineNumber":236,"sourceCode":"  cache.\n\n  Throughout the check, if source-to-cache has changed from the last check, it\n  also cleans up the invalidated cache early on.\n  \"\"\"\n  # TODO(BEAM-8335): we temporarily only cache replaceable unbounded sources.\n  # Add logic for other cacheable sources here when they are available.\n  has_cache = utils.has_unbounded_sources(user_pipeline)\n  if has_cache:\n    if not isinstance(ie.current_env().get_cache_manager(user_pipeline,\n                                                         create_if_absent=True),\n                      streaming_cache.StreamingCache):\n\n      file_based_cm = ie.current_env().get_cache_manager(user_pipeline)\n      cache_dir = file_based_cm._cache_dir\n      cache_root = ie.current_env().options.cache_root\n      if cache_root:\n        if cache_root.startswith('gs://'):\n          raise ValueError(\n              'GCS cache paths are not currently supported for '\n              'streaming pipelines.')\n        cache_dir = cache_root\n      ie.current_env().set_cache_manager(\n          streaming_cache.StreamingCache(\n              cache_dir,\n              is_cache_complete=is_cache_complete,\n              sample_resolution_sec=1.0,\n              saved_pcoders=file_based_cm._saved_pcoders),\n          user_pipeline)\n  return has_cache\n\n\ndef attempt_to_cancel_background_caching_job(user_pipeline):\n  \"\"\"Attempts to cancel background source recording job for a user-defined\n  pipeline.\n\n  If no background source recording job needs to be cancelled, NOOP. Otherwise,","sourceCodeStart":218,"sourceCodeEnd":254,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/interactive/background_caching_job.py#L218-L254","documentation":"Interactive Beam's background caching for streaming pipelines refuses GCS ('gs://') cache roots: recorded TestStream source and PCollections must be cached on a local filesystem path in streaming mode, so a gs:// ib.options.cache_root raises ValueError.","triggerScenarios":"Setting ib.options.cache_root = 'gs://bucket/path' and starting a background caching job (recording) for a streaming pipeline; is_background_caching_job_needed/_build_query_components detect the gs:// prefix and abort.","commonSituations":"Reusing the same cache_root configured for a Dataproc/remote interactive setup (which requires GCS) when running a streaming recording locally; copying GCS cache config from batch to streaming pipelines.","solutions":["Set ib.options.cache_root to a local directory (e.g. '/tmp/ib_cache') before recording the streaming pipeline.","Alternatively leave cache_root unset so Interactive Beam uses its default temp cache directory.","If GCS persistence is required, record/caches locally and copy files to GCS manually after the run."],"exampleFix":"// before\nib.options.cache_root = 'gs://my-bucket/ib-cache'\n// after\nib.options.cache_root = '/tmp/ib_cache'","handlingStrategy":"validation","validationCode":"import ib\nroot = ib.options.cache_root\nif pipeline_is_streaming and root and root.startswith('gs://'):\n    raise ValueError('Streaming pipelines need a local cache_root, not GCS.')","typeGuard":null,"tryCatchPattern":"try:\n    ib.options.cache_root = 'gs://bucket/c'\n    start_recording()\nexcept ValueError as e:\n    if 'GCS cache paths' in str(e):\n        ib.options.cache_root = '/tmp/ib_cache'\n        start_recording()","preventionTips":["Use local paths for cache_root in streaming recordings; reserve GCS for Dataproc setups.","Centralize cache_root configuration per pipeline mode (batch vs streaming)."],"tags":["apache-beam","interactive-beam","streaming","gcs","cache"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}