{"record":{"id":"c10cd223f64d7946","repo":"apache/beam","slug":"do-not-set-use-gbek-directly-pass-in-the-gbek-pipeline","errorCode":null,"errorMessage":"Do not set use_gbek directly, pass in the --gbek pipeline option with a valid secret instead.","messagePattern":"Do not set use_gbek directly, pass in the --gbek pipeline option with a valid secret instead\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/runners/dataflow/dataflow_runner.py","lineNumber":624,"sourceCode":"\n\ndef _check_and_add_missing_options(options):\n  # Type: (PipelineOptions) -> None\n\n  \"\"\"Validates and adds missing pipeline options depending on options set.\n\n  :param options: PipelineOptions for this pipeline.\n  \"\"\"\n  debug_options = options.view_as(DebugOptions)\n  dataflow_service_options = options.view_as(\n      GoogleCloudOptions).dataflow_service_options or []\n\n  # Add use_gbek to dataflow_service_options if gbek is set.\n  if options.view_as(SetupOptions).gbek:\n    if 'use_gbek' not in dataflow_service_options:\n      dataflow_service_options.append('use_gbek')\n  elif 'use_gbek' in dataflow_service_options:\n    raise ValueError(\n        'Do not set use_gbek directly, pass in the --gbek pipeline option '\n        'with a valid secret instead.')\n\n  _add_runner_v2_missing_options(options)\n\n  # Ensure that prime is specified as an experiment if specified as a dataflow\n  # service option\n  if 'enable_prime' in dataflow_service_options:\n    debug_options.add_experiment('enable_prime')\n  elif debug_options.lookup_experiment('enable_prime'):\n    dataflow_service_options.append('enable_prime')\n\n  options.view_as(\n      GoogleCloudOptions).dataflow_service_options = dataflow_service_options\n\n  sdk_location = options.view_as(SetupOptions).sdk_location\n  if 'dev' in beam.version.__version__ and sdk_location == 'default':\n    raise ValueError(","sourceCodeStart":606,"sourceCodeEnd":642,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L606-L642","documentation":"GBEK (Group By Key with an encrypted/hardened key) is enabled via the --gbek pipeline option (a secret reference), which internally adds the 'use_gbek' dataflow service option. Setting 'use_gbek' directly in dataflow_service_options bypasses the secret plumbing, so _check_and_add_missing_options rejects it.","triggerScenarios":"Passing --dataflow_service_options=use_gbek (or appending 'use_gbek' to GoogleCloudOptions.dataflow_service_options) without also setting the --gbek SetupOption; detected when 'use_gbek' is present in dataflow_service_options while options.view_as(SetupOptions).gbek is unset.","commonSituations":"Misreading documentation and enabling GBEK via service options instead of the --gbek flag; leftover service options from a config where --gbek was later removed.","solutions":["Remove 'use_gbek' from dataflow_service_options and instead pass --gbek=<secret> pipeline option.","If the option was set in code, delete the dataflow_service_options.append('use_gbek') line.","Confirm with: options.view_as(SetupOptions).gbek = 'gcp:secret:...' if configuring programmatically.","Re-run and verify the service option gets appended automatically from --gbek."],"exampleFix":"// before\noptions.view_as(GoogleCloudOptions).dataflow_service_options = ['use_gbek']\n// after\noptions.view_as(SetupOptions).gbek = 'projects/p/secrets/gbek-key'\n# use_gbek is added to dataflow_service_options automatically","handlingStrategy":"validation","validationCode":"dso = options.view_as(GoogleCloudOptions).dataflow_service_options or []\nassert 'use_gbek' not in dso or options.view_as(SetupOptions).gbek, 'set --gbek instead of use_gbek'","typeGuard":null,"tryCatchPattern":"try:\n    pipeline.run()\nexcept ValueError as e:\n    if 'use_gbek' in str(e):\n        opts.view_as(GoogleCloudOptions).dataflow_service_options = [o for o in (opts.view_as(GoogleCloudOptions).dataflow_service_options or []) if o != 'use_gbek']\n        opts.view_as(SetupOptions).gbek = 'gcp:secret:gbek-key'\n        pipeline.run()","preventionTips":["Enable GBEK only via the --gbek secret option","Never hand-edit dataflow_service_options for GBEK","Review config pipelines for leftover use_gbek entries after key rotation"],"tags":["python","apache-beam","dataflow","pipeline-options","gbek"],"backgroundTag":"conflicting-config-options","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}