apache/beam · error · ValueError

Materialized pipeline is not allocated for result cache.

Error message

Materialized pipeline is not allocated for result cache.

What it means

During pipeline materialization, a pipeline must first be registered in the in-process _pipeline_materialization_cache by _materialize_transform before results can be allocated. If _allocate_materialized_result is called with a pipeline that was never registered for the current process, this internal invariant ValueError fires.

Solutions

1. Use the materialization API only through its supported entry points (e.g. beam.Materialize on a running pipeline).
2. Ensure the pipeline is created and materialized in the same OS process.
3. Report a bug if it occurs in supported runner usage; this is an internal invariant.

try:
  result = beam.Materialize(pipeline, ...)
except ValueError as e:
  if 'not allocated for result cache' in str(e):
    raise RuntimeError('Pipeline must be materialized in the same process; recreate and materialize in one process') from e
  raise

When it happens

Trigger: Calling a materializing API (e.g. beam.Materialize / deferred materialization) with a pipeline object that never went through _materialize_transform in the same process, or after the cache was cleared.

Common situations: Using materialization across process boundaries (fork/spawn), mixing pipelines constructed in different processes, internal Beam bugs.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Unknown timing constant: " + timing
• A BigQuery table or a query must be specified
• A cluster_identifier should be Optional[Union[str…
• A context manager constructor (not a fully constructed…
• A has been supplied to the model handler, but the required…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0b2c544b42f16407. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/transforms/ptransform.py:166

}  # type: dict[tuple[int, int], dict[int, _MaterializedResult]]
_pipeline_materialization_lock = threading.Lock()


def _allocate_materialized_pipeline(pipeline):
  # type: (Pipeline) -> None
  pid = os.getpid()
  with _pipeline_materialization_lock:
    pipeline_id = id(pipeline)
    _pipeline_materialization_cache[(pid, pipeline_id)] = {}


def _allocate_materialized_result(pipeline):
  # type: (Pipeline) -> _MaterializedResult
  pid = os.getpid()
  with _pipeline_materialization_lock:
    pipeline_id = id(pipeline)
    if (pid, pipeline_id) not in _pipeline_materialization_cache:
      raise ValueError(
          'Materialized pipeline is not allocated for result '
          'cache.')
    result_id = len(_pipeline_materialization_cache[(pid, pipeline_id)])
    result = _MaterializedResult(pipeline_id, result_id)
    _pipeline_materialization_cache[(pid, pipeline_id)][result_id] = result
    return result


def _get_materialized_result(pipeline_id, result_id):
  # type: (int, int) -> _MaterializedResult
  pid = os.getpid()
  with _pipeline_materialization_lock:
    if (pid, pipeline_id) not in _pipeline_materialization_cache:
      raise Exception(
          'Materialization in out-of-process and remote runners is not yet '
          'supported.')
    return _pipeline_materialization_cache[(pid, pipeline_id)][result_id]

View on GitHub (pinned to 12126d8942)