{"record":{"id":"c645093bf2b3d84a","repo":"apache/beam","slug":"dataflow-can-only-execute-pipeline-steps-in-docker","errorCode":null,"errorMessage":"Dataflow can only execute pipeline steps in Docker environments. Received %r.","messagePattern":"Dataflow can only execute pipeline steps in Docker environments\\. Received %r\\.","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/runners/dataflow/internal/apiclient.py","lineNumber":231,"sourceCode":"    if self.worker_options.disk_provisioned_throughput_mibps is not None:\n      pool.disk_provisioned_throughput_mibps = (\n          self.worker_options.disk_provisioned_throughput_mibps)\n    if self.worker_options.zone:\n      pool.zone = self.worker_options.zone\n    if self.worker_options.network:\n      pool.network = self.worker_options.network\n    if self.worker_options.subnetwork:\n      pool.subnetwork = self.worker_options.subnetwork\n\n    # Setting worker pool sdk_harness_container_images option for supported\n    # Dataflow workers.\n    environments_to_use = self._get_environments_from_tranforms()\n\n    # Adding container images for other SDKs that may be needed for\n    # cross-language pipelines.\n    for id, environment in environments_to_use:\n      if environment.urn != common_urns.environments.DOCKER.urn:\n        raise Exception(\n            'Dataflow can only execute pipeline steps in Docker environments.'\n            ' Received %r.' % environment)\n      environment_payload = proto_utils.parse_Bytes(\n          environment.payload, beam_runner_api_pb2.DockerPayload)\n      container_image_url = environment_payload.container_image\n\n      container_image = dataflow.SdkHarnessContainerImage()\n      container_image.container_image = container_image_url\n      container_image.use_single_core_per_container = (\n          common_urns.protocols.MULTI_CORE_BUNDLE_PROCESSING.urn\n          not in environment.capabilities)\n      container_image.environment_id = id\n      for capability in environment.capabilities:\n        container_image.capabilities.append(capability)\n      pool.sdk_harness_container_images.append(container_image)\n\n    if not pool.sdk_harness_container_images:\n      pool.worker_harness_container_image = (","sourceCodeStart":213,"sourceCodeEnd":249,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py#L213-L249","documentation":"Exception raised in DataflowJob.__init__ (environment setup) when any pipeline environment is not the standard DOCKER environment. Dataflow workers only execute steps inside Docker containers, so non-Docker environments (e.g. custom SDK harness environments, process/external environments) are rejected.","triggerScenarios":"Submitting a cross-language pipeline or a pipeline whose components.environments contain an environment whose urn differs from beam:environments:docker:v1 — e.g. environments built for ProcessEnvironment or external runner environments.","commonSituations":"Cross-language transforms (Java/Go side of a Python pipeline) registering a non-Docker environment; constructing a beam_runner_api pipeline proto programmatically with a custom environment; running a portable pipeline proto intended for another runner against Dataflow.","solutions":["Ensure every environment in the pipeline uses the DOCKER urn, e.g. via beam.options.pipeline_options setup_options or default SDK environments","For cross-language pipelines, configure the expansion service to produce Docker environments with proper container_image","If building pipeline protos manually, set environment urn to 'beam:environments:docker:v1' with a DockerPayload container_image"],"exampleFix":"# before\nfrom apache_beam.portability.api import beam_runner_api_pb2\nenv = beam_runner_api_pb2.Environment(urn='beam:environments:process:v1')\n# after\nenv = beam_runner_api_pb2.Environment(\n    urn='beam:environments:docker:v1',\n    payload=proto_utils.to_Bytes(\n        beam_runner_api_pb2.DockerPayload(container_image='apache/beam_python3.11_sdk:latest')))","handlingStrategy":"validation","validationCode":"from apache_beam.portability import common_urns\nfor env in pipeline.proto.components.environments.values():\n    if env.urn != common_urns.environments.DOCKER.urn:\n        raise ValueError('non-Docker environment: %s' % env.urn)","typeGuard":"def is_docker_env(env) -> bool:\n    return env.urn == 'beam:environments:docker:v1'","tryCatchPattern":"try:\n    job = Job(pipeline, options)\nexcept Exception as e:\n    if 'Docker environments' in str(e):\n        print('Pipeline has non-Docker environments; fix expansion service config')\n    raise","preventionTips":["Use default SDK environments for all steps","Configure cross-language expansion services to emit Docker environments with proper container images","Inspect pipeline proto environments before submitting Dataflow jobs"],"tags":["dataflow","docker","environment","cross-language"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}