apache/beam · error · Exception

Materialization in out-of-process and remote runners is not…

Error message

Materialization in out-of-process and remote runners is not yet supported.

What it means

_get_materialized_result looks up materialized results from a per-process cache keyed by (pid, pipeline_id). Out-of-process or remote runners cannot access this cache, so if the key is absent the error explains that materialization is not yet supported for those runners.

Solutions

1. Run the pipeline on the DirectRunner (in-process) when using materialization.
2. Remove/restructure materialization usage for distributed runners.
3. Check the Beam roadmap/issue tracker for materialization support on your runner and upgrade when available.

Example fix

// before
with beam.Pipeline(runner='DataflowRunner') as p:  # materialization unsupported
  _ = beam.Materialize(...)
// after
with beam.Pipeline(runner='DirectRunner') as p:
  _ = beam.Materialize(...)

runner_ok = getattr(pipeline.runner, 'is_eager', False) or type(pipeline.runner).__name__ == 'DirectRunner'
assert runner_ok, "materialization requires in-process runner"

try:
  mat = beam.Materialize(pipeline, ...)
except Exception as e:
  if 'Materialization' in str(e):
    mat = pipeline | fallback_transform  # non-materializing path
  else:
    raise

When it happens

Trigger: Using pipeline materialization (beam.Materialize / materialized PTransform results) with a runner that executes transforms in other processes (e.g. Dataflow, Flink, or multiprocessing-based runners).

Common situations: Submitting a pipeline with materialization to Dataflow or a portable runner; running under spawn-based multiprocessing where the cache is empty in the worker.

Related errors

• A BigQuery table or a query must be specified
• A cluster_identifier should be Optional[Union[str…
• A context manager constructor (not a fully constructed…
• A has been supplied to the model handler, but the required…
• A pubsub message attribute key must not exceed 256 bytes.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a07dfbb4e2ff3405. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/transforms/ptransform.py:180

  pid = os.getpid()
  with _pipeline_materialization_lock:
    pipeline_id = id(pipeline)
    if (pid, pipeline_id) not in _pipeline_materialization_cache:
      raise ValueError(
          'Materialized pipeline is not allocated for result '
          'cache.')
    result_id = len(_pipeline_materialization_cache[(pid, pipeline_id)])
    result = _MaterializedResult(pipeline_id, result_id)
    _pipeline_materialization_cache[(pid, pipeline_id)][result_id] = result
    return result


def _get_materialized_result(pipeline_id, result_id):
  # type: (int, int) -> _MaterializedResult
  pid = os.getpid()
  with _pipeline_materialization_lock:
    if (pid, pipeline_id) not in _pipeline_materialization_cache:
      raise Exception(
          'Materialization in out-of-process and remote runners is not yet '
          'supported.')
    return _pipeline_materialization_cache[(pid, pipeline_id)][result_id]


def _release_materialized_pipeline(pipeline):
  # type: (Pipeline) -> None
  pid = os.getpid()
  with _pipeline_materialization_lock:
    pipeline_id = id(pipeline)
    del _pipeline_materialization_cache[(pid, pipeline_id)]


class _MaterializedResult(object):
  def __init__(self, pipeline_id, result_id):
    # type: (int, int) -> None
    self._pipeline_id = pipeline_id
    self._result_id = result_id

View on GitHub (pinned to 12126d8942)