apache/beam · error · ValueError

GCS cache paths are not currently supported for streaming…

Error message

GCS cache paths are not currently supported for streaming pipelines.

What it means

Interactive Beam's background caching for streaming pipelines refuses GCS ('gs://') cache roots: recorded TestStream source and PCollections must be cached on a local filesystem path in streaming mode, so a gs:// ib.options.cache_root raises ValueError.

Solutions

1. Set ib.options.cache_root to a local directory (e.g. '/tmp/ib_cache') before recording the streaming pipeline.
2. Alternatively leave cache_root unset so Interactive Beam uses its default temp cache directory.
3. If GCS persistence is required, record/caches locally and copy files to GCS manually after the run.

Example fix

// before
ib.options.cache_root = 'gs://my-bucket/ib-cache'
// after
ib.options.cache_root = '/tmp/ib_cache'

import ib
root = ib.options.cache_root
if pipeline_is_streaming and root and root.startswith('gs://'):
    raise ValueError('Streaming pipelines need a local cache_root, not GCS.')

try:
    ib.options.cache_root = 'gs://bucket/c'
    start_recording()
except ValueError as e:
    if 'GCS cache paths' in str(e):
        ib.options.cache_root = '/tmp/ib_cache'
        start_recording()

When it happens

Trigger: Setting ib.options.cache_root = 'gs://bucket/path' and starting a background caching job (recording) for a streaming pipeline; is_background_caching_job_needed/_build_query_components detect the gs:// prefix and abort.

Common situations: Reusing the same cache_root configured for a Dataproc/remote interactive setup (which requires GCS) when running a streaming recording locally; copying GCS cache config from batch to streaming pipelines.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Timed out waiting for cache file for PCollection
• ib.options.cache_root needs to be a Cloud Storage Bucket to…
• Unsupported cache format
• Values given to streaming cache should be either…
• An unsupported type of cache was passed in. Received

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f8ddd94aeec3bcde. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/interactive/background_caching_job.py:236

  cache.

  Throughout the check, if source-to-cache has changed from the last check, it
  also cleans up the invalidated cache early on.
  """
  # TODO(BEAM-8335): we temporarily only cache replaceable unbounded sources.
  # Add logic for other cacheable sources here when they are available.
  has_cache = utils.has_unbounded_sources(user_pipeline)
  if has_cache:
    if not isinstance(ie.current_env().get_cache_manager(user_pipeline,
                                                         create_if_absent=True),
                      streaming_cache.StreamingCache):

      file_based_cm = ie.current_env().get_cache_manager(user_pipeline)
      cache_dir = file_based_cm._cache_dir
      cache_root = ie.current_env().options.cache_root
      if cache_root:
        if cache_root.startswith('gs://'):
          raise ValueError(
              'GCS cache paths are not currently supported for '
              'streaming pipelines.')
        cache_dir = cache_root
      ie.current_env().set_cache_manager(
          streaming_cache.StreamingCache(
              cache_dir,
              is_cache_complete=is_cache_complete,
              sample_resolution_sec=1.0,
              saved_pcoders=file_based_cm._saved_pcoders),
          user_pipeline)
  return has_cache


def attempt_to_cancel_background_caching_job(user_pipeline):
  """Attempts to cancel background source recording job for a user-defined
  pipeline.

  If no background source recording job needs to be cancelled, NOOP. Otherwise,

View on GitHub (pinned to 12126d8942)