apache/beam · error · Exception

Dataflow can only execute pipeline steps in Docker…

Error message

Dataflow can only execute pipeline steps in Docker environments. Received %r.

What it means

Exception raised in DataflowJob.__init__ (environment setup) when any pipeline environment is not the standard DOCKER environment. Dataflow workers only execute steps inside Docker containers, so non-Docker environments (e.g. custom SDK harness environments, process/external environments) are rejected.

Solutions

1. Ensure every environment in the pipeline uses the DOCKER urn, e.g. via beam.options.pipeline_options setup_options or default SDK environments
2. For cross-language pipelines, configure the expansion service to produce Docker environments with proper container_image
3. If building pipeline protos manually, set environment urn to 'beam:environments:docker:v1' with a DockerPayload container_image

Example fix

# before
from apache_beam.portability.api import beam_runner_api_pb2
env = beam_runner_api_pb2.Environment(urn='beam:environments:process:v1')
# after
env = beam_runner_api_pb2.Environment(
    urn='beam:environments:docker:v1',
    payload=proto_utils.to_Bytes(
        beam_runner_api_pb2.DockerPayload(container_image='apache/beam_python3.11_sdk:latest')))

from apache_beam.portability import common_urns
for env in pipeline.proto.components.environments.values():
    if env.urn != common_urns.environments.DOCKER.urn:
        raise ValueError('non-Docker environment: %s' % env.urn)

def is_docker_env(env) -> bool:
    return env.urn == 'beam:environments:docker:v1'

try:
    job = Job(pipeline, options)
except Exception as e:
    if 'Docker environments' in str(e):
        print('Pipeline has non-Docker environments; fix expansion service config')
    raise

When it happens

Trigger: Submitting a cross-language pipeline or a pipeline whose components.environments contain an environment whose urn differs from beam:environments:docker:v1 — e.g. environments built for ProcessEnvironment or external runner environments.

Common situations: Cross-language transforms (Java/Go side of a Python pipeline) registering a non-Docker environment; constructing a beam_runner_api pipeline proto programmatically with a custom environment; running a portable pipeline proto intended for another runner against Dataflow.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• 'No available provider for type %r at
• No proto encoding for PaneInfoCoder, always part of…
• unsupported artifact type
• Unsupported environment
• A non-standard version of Beam SDK detected

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c645093bf2b3d84a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/dataflow/internal/apiclient.py:231

    if self.worker_options.disk_provisioned_throughput_mibps is not None:
      pool.disk_provisioned_throughput_mibps = (
          self.worker_options.disk_provisioned_throughput_mibps)
    if self.worker_options.zone:
      pool.zone = self.worker_options.zone
    if self.worker_options.network:
      pool.network = self.worker_options.network
    if self.worker_options.subnetwork:
      pool.subnetwork = self.worker_options.subnetwork

    # Setting worker pool sdk_harness_container_images option for supported
    # Dataflow workers.
    environments_to_use = self._get_environments_from_tranforms()

    # Adding container images for other SDKs that may be needed for
    # cross-language pipelines.
    for id, environment in environments_to_use:
      if environment.urn != common_urns.environments.DOCKER.urn:
        raise Exception(
            'Dataflow can only execute pipeline steps in Docker environments.'
            ' Received %r.' % environment)
      environment_payload = proto_utils.parse_Bytes(
          environment.payload, beam_runner_api_pb2.DockerPayload)
      container_image_url = environment_payload.container_image

      container_image = dataflow.SdkHarnessContainerImage()
      container_image.container_image = container_image_url
      container_image.use_single_core_per_container = (
          common_urns.protocols.MULTI_CORE_BUNDLE_PROCESSING.urn
          not in environment.capabilities)
      container_image.environment_id = id
      for capability in environment.capabilities:
        container_image.capabilities.append(capability)
      pool.sdk_harness_container_images.append(container_image)

    if not pool.sdk_harness_container_images:
      pool.worker_harness_container_image = (

View on GitHub (pinned to 12126d8942)