apache/beam · error · ValueError

SDK Docker container image has to be a non-empty string

Error message

SDK Docker container image has to be a non-empty string

What it means

Raised by the Beam Dataflow runner when the SDK harness Docker container image resolves to an empty or falsy string after applying environment overrides. Dataflow workers need a valid container image to launch the SDK harness, so an empty image would produce a broken job. The library throws a ValueError early at pipeline submission to fail fast instead of at worker startup.

Solutions

1. Set --sdk_container_image (or the equivalent container_image option) to a non-empty valid image URI, e.g. 'gcr.io/cloud-dataflow/v1beta3/beam_python3.11_sdk:latest'.
2. If overriding via a custom environment, ensure docker_payload.container_image is populated before the payload is serialized.
3. Check whether a config/template variable feeding sdk_container_image is interpolated to an empty string and give it a default.

Example fix

// before
python -m my_pipeline --runner DataflowRunner --sdk_container_image=""
// after
python -m my_pipeline --runner DataflowRunner --sdk_container_image="gcr.io/cloud-dataflow/v1beta3/beam_python3.11_sdk:latest"

image = getattr(pipeline_options, 'sdk_container_image', None)
if not image:
  raise ValueError('sdk_container_image must be set to a non-empty string before submitting to Dataflow')

def has_container_image(opts) -> bool:
    img = getattr(opts, 'sdk_container_image', None)
    return isinstance(img, str) and img.strip() != ''

try:
    result = pipeline.run()
except ValueError as e:
    if 'container image' in str(e):
        logging.error('Set --sdk_container_image to a valid image URI: %s', e)
    raise

When it happens

Trigger: Calling Pipeline.run() against DataflowRunner where a DockerPayload environment override exists but the resulting container image is empty — e.g. pipeline option --sdk_container_image set to an empty string, or a custom environment override setting container_image to None/''.

Common situations: CI config rendering an empty --sdk_container_image variable; templates with an unset placeholder for the container image; custom _apply_sdk_environment_overrides hooks passing a blank image.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• buffer_sec must be >= 0, got
• Can not query metrics. Job id is unknown.
• "Cannot specify 'callable' with 'path' and 'name' for…
• Chain at missing transforms property.
• Coder for the GroupByKey operation

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a4bc3287b7e6745e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/dataflow/internal/apiclient.py:844

      docker_payload = proto_utils.parse_Bytes(
          environment.payload, beam_runner_api_pb2.DockerPayload)
      overridden = False
      new_container_image = docker_payload.container_image
      for pattern, override in sdk_overrides.items():
        new_container_image = re.sub(pattern, override, new_container_image)
        if new_container_image != docker_payload.container_image:
          overridden = True

      # Container of the current (Python) SDK is overridden separately, hence
      # not updated here.
      if (is_apache_beam_container(new_container_image) and not overridden and
          new_container_image != current_sdk_container_image):
        new_container_image = (
            DataflowApplicationClient._update_container_image_for_dataflow(
                docker_payload.container_image))

      if not new_container_image:
        raise ValueError(
            'SDK Docker container image has to be a non-empty string')

      new_payload = copy(docker_payload)
      new_payload.container_image = new_container_image
      environment.payload = new_payload.SerializeToString()

  def create_job_description(self, job: Job):
    """Creates a job described by the workflow proto."""
    DataflowApplicationClient._apply_sdk_environment_overrides(
        job.proto_pipeline, self._sdk_image_overrides, job.options)

    # Stage other resources for the SDK harness
    resources = self._stage_resources(job.proto_pipeline, job.options)

    # Stage proto pipeline.
    serialized_pipeline = job.proto_pipeline.SerializeToString()
    pipeline_proto_hash = hashlib.sha256(serialized_pipeline).hexdigest()

View on GitHub (pinned to 12126d8942)