apache/beam · error · ValueError

PCollection not available, please run the pipeline.

Error message

PCollection not available, please run the pipeline.

What it means

CacheManager-backed PCollection readers (used by ib.collect / recording get()) materialize elements from a 'full' cache written when the pipeline ran. read() checks cache_manager.exists('full', key); if the cached data for this PCollection's key is missing it raises this ValueError instead of silently returning empty data. It means the pipeline containing this PCollection has not been executed (or its cache was cleared) in the current interactive session.

Source

Thrown at sdks/python/apache_beam/runners/interactive/interactive_runner.py:353

    WindowedValues. Otherwise, return the element as itself.
    """
    return list(self.read(pcoll, include_window_info))

  def read(self, pcoll, include_window_info=False):
    """Reads the PCollection one element at a time from cache.

    If include_window_info is True, then returns the elements as
    WindowedValues. Otherwise, return the element as itself.
    """
    key = self._pipeline_instrument.cache_key(pcoll)
    cache_manager = ie.current_env().get_cache_manager(
        self._pipeline_instrument.user_pipeline)
    if key and cache_manager.exists('full', key):
      coder = cache_manager.load_pcoder('full', key)
      reader, _ = cache_manager.read('full', key)
      return to_element_list(reader, coder, include_window_info)
    else:
      raise ValueError('PCollection not available, please run the pipeline.')

  def cancel(self):
    self._underlying_result.cancel()

View on GitHub (pinned to 12126d8942)

Solutions

1. Run (or re-run) the pipeline with the InteractiveRunner so the PCollection is recorded into the cache, then call get/collect again.
2. Re-execute ib.show/ib.compute on the pcoll to create a fresh recording, then collect.
3. Check your cache_root directory for the pcoll's cache files to confirm the recording exists.
4. Don't change ib.options.cache_root between defining and collecting a pcoll.
5. Re-run all upstream cells after a kernel restart so watches and recordings are re-established.

Example fix

// before: pc = p | beam.Map(lambda x: x); ib.collect(pc)  # pipeline never ran | // after: pc = p | beam.Map(lambda x: x); ib.show(pc)  # runs/records; then ib.collect(pc)

def has_recording(pcoll): return ib.current_env().get_recording_manager(pcoll.pipeline, create_if_absent=False) is not None; assert has_recording(pc)  # else run ib.show/ib.compute first

def is_recorded(pcoll): return ib.current_env().get_recording_manager(pcoll.pipeline, create_if_absent=False) is not None

try: rows = ib.collect(pc) | except ValueError as e: (ib.compute(pc); rows = ib.collect(pc)) if 'PCollection not available' in str(e) else raise

When it happens

Trigger: Calling ib.collect(pcoll) (which routes through RecordingManager.get → read) before running the pipeline that produces pcoll; the cache directory was deleted or cache_root changed between runs; the pipeline ran but this PCollection was outside the recorded/instrumented set; kernel restart lost the recording while cache files were cleaned.

Common situations: Notebooks: editing a pcoll definition after the last run then collecting without re-running; cache_root pointing to ephemeral storage cleared between sessions; using ib.collect on a pcoll from a pipeline built with a non-interactive runner so nothing was recorded; cache cleanup removed files.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• The given pcoll %s is not a dict, an iterable or a PCollecti
• All PCollections must belong to the same pipeline.
• The given pcoll {pcoll_container} is not a dict, an iterable
• The beam_sql magic tries to query PCollections from multiple
• cache_root GCS bucket path is invalid.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a41d5ddeaf5621f1. Report an issue: GitHub.