apache/beam · error · ValueError

You are submitting a pipeline with Apache Beam Python SDK

Error message

You are submitting a pipeline with Apache Beam Python SDK %s. When launching Dataflow jobs with an unreleased (dev) SDK, please provide an SDK distribution in the --sdk_location option to use a consistent SDK version at pipeline submission and runtime. To ignore this error and use an SDK preinstalled in the default Dataflow dev runtime environment or in a custom container image, use --sdk_location=container.

What it means

Pipelines built with an unreleased (dev) Beam SDK cannot rely on the Dataflow default runtime image, because the preinstalled runtime SDK would not match the dev SDK version. run-time checks in _check_and_add_missing_options raise this ValueError when beam.version contains 'dev' and sdk_location is left as 'default'.

Solutions

Pass --sdk_location pointing to your built wheel or sdist, e.g. --sdk_location=dist/apache_beam-2.x.dev0-cp310-...whl.
Or explicitly opt into the container runtime with --sdk_location=container if the image has the matching SDK.
Or install a released apache-beam version instead of a dev build before submitting.
For local tests use the DirectRunner, which doesn't require --sdk_location.

Example fix

// before
# dev SDK, no sdk_location
python my_pipeline.py --runner DataflowRunner ...
// after
python my_pipeline.py --runner DataflowRunner --sdk_location=dist/apache_beam-2.61.0.dev0-cp310-cp310-manylinux1_x86_64.whl ...

Defensive patterns

Strategy: validation

Validate before calling

import apache_beam
if 'dev' in apache_beam.__version__ and (opts.view_as(SetupOptions).sdk_location or 'default') == 'default':
    raise SystemExit('dev SDK requires --sdk_location or --sdk_location=container')

Try / catch

try:
    pipeline.run()
except ValueError as e:
    if 'unreleased (dev) SDK' in str(e):
        opts.view_as(SetupOptions).sdk_location = 'container'
        pipeline.run()

Prevention

Always pass --sdk_location when building/testing dev SDKs
Use released apache-beam versions for production submissions
Add a CI guard that blocks Dataflow submissions from dev installs without sdk_location

When it happens

Trigger: Submitting a Dataflow job from a source install / nightly / locally-built apache-beam (version string contains 'dev') without setting --sdk_location, so it stays 'default'.

Common situations: Testing Beam source changes against Dataflow; using a nightly apache-beam wheel; forgetting --sdk_location after cloning the Beam repo.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1949f3c15803f09b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/dataflow/dataflow_runner.py:642

    raise ValueError(
        'Do not set use_gbek directly, pass in the --gbek pipeline option '
        'with a valid secret instead.')

  _add_runner_v2_missing_options(options)

  # Ensure that prime is specified as an experiment if specified as a dataflow
  # service option
  if 'enable_prime' in dataflow_service_options:
    debug_options.add_experiment('enable_prime')
  elif debug_options.lookup_experiment('enable_prime'):
    dataflow_service_options.append('enable_prime')

  options.view_as(
      GoogleCloudOptions).dataflow_service_options = dataflow_service_options

  sdk_location = options.view_as(SetupOptions).sdk_location
  if 'dev' in beam.version.__version__ and sdk_location == 'default':
    raise ValueError(
        "You are submitting a pipeline with Apache Beam Python SDK "
        f"{beam.version.__version__}. "
        "When launching Dataflow jobs with an unreleased (dev) SDK, "
        "please provide an SDK distribution in the --sdk_location option "
        "to use a consistent SDK version at "
        "pipeline submission and runtime. To ignore this error and use "
        "an SDK preinstalled in the default Dataflow dev runtime environment "
        "or in a custom container image, use --sdk_location=container.")


def _check_and_add_missing_streaming_options(options):
  # Type: (PipelineOptions) -> None

  """Validates and adds missing pipeline options depending on options set.

  Must be called after it has been determined whether we're running in
  streaming mode.

View on GitHub (pinned to 12126d8942)