{"record":{"id":"1a388d458ca92c70","repo":"apache/beam","slug":"streaming-python-direct-runner-does-not-support-cross","errorCode":null,"errorMessage":"Streaming Python direct runner does not support cross-language pipelines.Please use other runners such as FlinkRunner, DataflowRunner, or PrismRunner.","messagePattern":"Streaming Python direct runner does not support cross-language pipelines\\.Please use other runners such as FlinkRunner, DataflowRunner, or PrismRunner\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/runners/direct/direct_runner.py","lineNumber":570,"sourceCode":"  def run_pipeline(self, pipeline, options):\n    \"\"\"Execute the entire pipeline and returns an DirectPipelineResult.\"\"\"\n\n    # TODO: Move imports to top. Pipeline <-> Runner dependency cause problems\n    # with resolving imports when they are at top.\n    # pylint: disable=wrong-import-position\n    from apache_beam.pipeline import PipelineVisitor\n    from apache_beam.runners.direct.consumer_tracking_pipeline_visitor import ConsumerTrackingPipelineVisitor\n    from apache_beam.runners.direct.evaluation_context import EvaluationContext\n    from apache_beam.runners.direct.executor import Executor\n    from apache_beam.runners.direct.transform_evaluator import TransformEvaluatorRegistry\n    from apache_beam.testing.test_stream import TestStream\n    from apache_beam.transforms.external import ExternalTransform\n\n    class VerifyNoCrossLanguageTransforms(PipelineVisitor):\n      \"\"\"Visitor determining whether a Pipeline uses a TestStream.\"\"\"\n      def visit_transform(self, applied_ptransform):\n        if isinstance(applied_ptransform.transform, ExternalTransform):\n          raise RuntimeError(\n              \"Streaming Python direct runner \"\n              \"does not support cross-language pipelines.\"\n              \"Please use other runners such as FlinkRunner, \"\n              \"DataflowRunner, or PrismRunner.\")\n\n    pipeline.visit(VerifyNoCrossLanguageTransforms())\n\n    # If the TestStream I/O is used, use a mock test clock.\n    class TestStreamUsageVisitor(PipelineVisitor):\n      \"\"\"Visitor determining whether a Pipeline uses a TestStream.\"\"\"\n      def __init__(self):\n        self.uses_test_stream = False\n\n      def visit_transform(self, applied_ptransform):\n        if isinstance(applied_ptransform.transform, TestStream):\n          self.uses_test_stream = True\n\n    visitor = TestStreamUsageVisitor()","sourceCodeStart":552,"sourceCodeEnd":588,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/direct/direct_runner.py#L552-L588","documentation":"The streaming DirectRunner rejects cross-language pipelines: its visitor raises RuntimeError when it encounters an ExternalTransform (xlang transform) in the pipeline graph. Cross-language transforms need runner-side expansion services that the Python direct streaming path doesn't support, so the failure happens up front with suggested alternative runners.","triggerScenarios":"Running a streaming pipeline on the DirectRunner that contains cross-language transforms such as Kafka IO via Java expansion (ExternalTransform).","commonSituations":"Prototyping a Kafka/Kinesis/xlang pipeline locally with --streaming before deploying to a real runner; tests that use cross-language IO with the direct runner.","solutions":["Switch to a runner that supports cross-language pipelines: FlinkRunner, DataflowRunner, or PrismRunner.","Test non-xlang parts of the pipeline locally and exercise xlang IO only on a supporting runner.","Remove the ExternalTransform for local tests, mocking its input/output."],"exampleFix":"// before\n# streaming xlang pipeline locally\nwith beam.Pipeline(runner='DirectRunner', options=streaming_opts) as p:\n  p | KafkaIO.read(...)   # ExternalTransform -> RuntimeError\n// after\nwith beam.Pipeline(runner='FlinkRunner', options=flink_opts) as p:\n  p | KafkaIO.read(...)","handlingStrategy":"validation","validationCode":"from apache_beam.transforms.external import ExternalTransform\ndef uses_xlang(pipeline) -> bool:\n    found = []\n    pipeline.apply_visitor(lambda t: found.append(t) if isinstance(getattr(t, 'transform', None), ExternalTransform) else None)\n    return bool(found)","typeGuard":null,"tryCatchPattern":"try:\n    result = pipeline.run()\nexcept RuntimeError as e:\n    if 'cross-language' in str(e):\n        sys.exit('Use FlinkRunner/DataflowRunner/PrismRunner for xlang pipelines')\n    raise","preventionTips":["Reserve DirectRunner for pure-Python batch pipelines.","Test cross-language IO only on runners with expansion support.","Centralize runner selection in config so local runs can be swapped easily."],"tags":["python","apache-beam","direct-runner","cross-language","streaming"],"backgroundTag":"operation-not-supported","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}