{"record":{"id":"a07dfbb4e2ff3405","repo":"apache/beam","slug":"materialization-in-out-of-process-and-remote-runners-is-not","errorCode":null,"errorMessage":"Materialization in out-of-process and remote runners is not yet supported.","messagePattern":"Materialization in out-of-process and remote runners is not yet supported\\.","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/transforms/ptransform.py","lineNumber":180,"sourceCode":"  pid = os.getpid()\n  with _pipeline_materialization_lock:\n    pipeline_id = id(pipeline)\n    if (pid, pipeline_id) not in _pipeline_materialization_cache:\n      raise ValueError(\n          'Materialized pipeline is not allocated for result '\n          'cache.')\n    result_id = len(_pipeline_materialization_cache[(pid, pipeline_id)])\n    result = _MaterializedResult(pipeline_id, result_id)\n    _pipeline_materialization_cache[(pid, pipeline_id)][result_id] = result\n    return result\n\n\ndef _get_materialized_result(pipeline_id, result_id):\n  # type: (int, int) -> _MaterializedResult\n  pid = os.getpid()\n  with _pipeline_materialization_lock:\n    if (pid, pipeline_id) not in _pipeline_materialization_cache:\n      raise Exception(\n          'Materialization in out-of-process and remote runners is not yet '\n          'supported.')\n    return _pipeline_materialization_cache[(pid, pipeline_id)][result_id]\n\n\ndef _release_materialized_pipeline(pipeline):\n  # type: (Pipeline) -> None\n  pid = os.getpid()\n  with _pipeline_materialization_lock:\n    pipeline_id = id(pipeline)\n    del _pipeline_materialization_cache[(pid, pipeline_id)]\n\n\nclass _MaterializedResult(object):\n  def __init__(self, pipeline_id, result_id):\n    # type: (int, int) -> None\n    self._pipeline_id = pipeline_id\n    self._result_id = result_id","sourceCodeStart":162,"sourceCodeEnd":198,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/transforms/ptransform.py#L162-L198","documentation":"_get_materialized_result looks up materialized results from a per-process cache keyed by (pid, pipeline_id). Out-of-process or remote runners cannot access this cache, so if the key is absent the error explains that materialization is not yet supported for those runners.","triggerScenarios":"Using pipeline materialization (beam.Materialize / materialized PTransform results) with a runner that executes transforms in other processes (e.g. Dataflow, Flink, or multiprocessing-based runners).","commonSituations":"Submitting a pipeline with materialization to Dataflow or a portable runner; running under spawn-based multiprocessing where the cache is empty in the worker.","solutions":["Run the pipeline on the DirectRunner (in-process) when using materialization.","Remove/restructure materialization usage for distributed runners.","Check the Beam roadmap/issue tracker for materialization support on your runner and upgrade when available."],"exampleFix":"// before\nwith beam.Pipeline(runner='DataflowRunner') as p:  # materialization unsupported\n  _ = beam.Materialize(...)\n// after\nwith beam.Pipeline(runner='DirectRunner') as p:\n  _ = beam.Materialize(...)","handlingStrategy":"fallback","validationCode":"runner_ok = getattr(pipeline.runner, 'is_eager', False) or type(pipeline.runner).__name__ == 'DirectRunner'\nassert runner_ok, \"materialization requires in-process runner\"","typeGuard":null,"tryCatchPattern":"try:\n  mat = beam.Materialize(pipeline, ...)\nexcept Exception as e:\n  if 'Materialization' in str(e):\n    mat = pipeline | fallback_transform  # non-materializing path\n  else:\n    raise","preventionTips":["Use DirectRunner when testing materialization","Avoid beam.Materialize in Dataflow/Flink jobs until supported"],"tags":["python","apache-beam","runner-limitation"],"backgroundTag":"operation-not-supported","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}