{"record":{"id":"1949f3c15803f09b","repo":"apache/beam","slug":"you-are-submitting-a-pipeline-with-apache-beam-python-sdk-s","errorCode":null,"errorMessage":"You are submitting a pipeline with Apache Beam Python SDK %s. When launching Dataflow jobs with an unreleased (dev) SDK, please provide an SDK distribution in the --sdk_location option to use a consistent SDK version at pipeline submission and runtime. To ignore this error and use an SDK preinstalled in the default Dataflow dev runtime environment or in a custom container image, use --sdk_location=container.","messagePattern":"You are submitting a pipeline with Apache Beam Python SDK (.+?)\\. When launching Dataflow jobs with an unreleased \\(dev\\) SDK, please provide an SDK distribution in the --sdk_location option to use a consistent SDK version at pipeline submission and runtime\\. To ignore this error and use an SDK preinstalled in the default Dataflow dev runtime environment or in a custom container image, use --sdk_location=container\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/runners/dataflow/dataflow_runner.py","lineNumber":642,"sourceCode":"    raise ValueError(\n        'Do not set use_gbek directly, pass in the --gbek pipeline option '\n        'with a valid secret instead.')\n\n  _add_runner_v2_missing_options(options)\n\n  # Ensure that prime is specified as an experiment if specified as a dataflow\n  # service option\n  if 'enable_prime' in dataflow_service_options:\n    debug_options.add_experiment('enable_prime')\n  elif debug_options.lookup_experiment('enable_prime'):\n    dataflow_service_options.append('enable_prime')\n\n  options.view_as(\n      GoogleCloudOptions).dataflow_service_options = dataflow_service_options\n\n  sdk_location = options.view_as(SetupOptions).sdk_location\n  if 'dev' in beam.version.__version__ and sdk_location == 'default':\n    raise ValueError(\n        \"You are submitting a pipeline with Apache Beam Python SDK \"\n        f\"{beam.version.__version__}. \"\n        \"When launching Dataflow jobs with an unreleased (dev) SDK, \"\n        \"please provide an SDK distribution in the --sdk_location option \"\n        \"to use a consistent SDK version at \"\n        \"pipeline submission and runtime. To ignore this error and use \"\n        \"an SDK preinstalled in the default Dataflow dev runtime environment \"\n        \"or in a custom container image, use --sdk_location=container.\")\n\n\ndef _check_and_add_missing_streaming_options(options):\n  # Type: (PipelineOptions) -> None\n\n  \"\"\"Validates and adds missing pipeline options depending on options set.\n\n  Must be called after it has been determined whether we're running in\n  streaming mode.\n","sourceCodeStart":624,"sourceCodeEnd":660,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L624-L660","documentation":"Pipelines built with an unreleased (dev) Beam SDK cannot rely on the Dataflow default runtime image, because the preinstalled runtime SDK would not match the dev SDK version. run-time checks in _check_and_add_missing_options raise this ValueError when beam.version contains 'dev' and sdk_location is left as 'default'.","triggerScenarios":"Submitting a Dataflow job from a source install / nightly / locally-built apache-beam (version string contains 'dev') without setting --sdk_location, so it stays 'default'.","commonSituations":"Testing Beam source changes against Dataflow; using a nightly apache-beam wheel; forgetting --sdk_location after cloning the Beam repo.","solutions":["Pass --sdk_location pointing to your built wheel or sdist, e.g. --sdk_location=dist/apache_beam-2.x.dev0-cp310-...whl.","Or explicitly opt into the container runtime with --sdk_location=container if the image has the matching SDK.","Or install a released apache-beam version instead of a dev build before submitting.","For local tests use the DirectRunner, which doesn't require --sdk_location."],"exampleFix":"// before\n# dev SDK, no sdk_location\npython my_pipeline.py --runner DataflowRunner ...\n// after\npython my_pipeline.py --runner DataflowRunner --sdk_location=dist/apache_beam-2.61.0.dev0-cp310-cp310-manylinux1_x86_64.whl ...","handlingStrategy":"validation","validationCode":"import apache_beam\nif 'dev' in apache_beam.__version__ and (opts.view_as(SetupOptions).sdk_location or 'default') == 'default':\n    raise SystemExit('dev SDK requires --sdk_location or --sdk_location=container')","typeGuard":null,"tryCatchPattern":"try:\n    pipeline.run()\nexcept ValueError as e:\n    if 'unreleased (dev) SDK' in str(e):\n        opts.view_as(SetupOptions).sdk_location = 'container'\n        pipeline.run()","preventionTips":["Always pass --sdk_location when building/testing dev SDKs","Use released apache-beam versions for production submissions","Add a CI guard that blocks Dataflow submissions from dev installs without sdk_location"],"tags":["python","apache-beam","dataflow","dev-sdk","pipeline-options"],"backgroundTag":"invalid-config-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}