apache/beam · error · ValueError

Do not set use_gbek directly, pass in the --gbek pipeline…

Error message

Do not set use_gbek directly, pass in the --gbek pipeline option with a valid secret instead.

What it means

GBEK (Group By Key with an encrypted/hardened key) is enabled via the --gbek pipeline option (a secret reference), which internally adds the 'use_gbek' dataflow service option. Setting 'use_gbek' directly in dataflow_service_options bypasses the secret plumbing, so _check_and_add_missing_options rejects it.

Solutions

  1. Remove 'use_gbek' from dataflow_service_options and instead pass --gbek=<secret> pipeline option.
  2. If the option was set in code, delete the dataflow_service_options.append('use_gbek') line.
  3. Confirm with: options.view_as(SetupOptions).gbek = 'gcp:secret:...' if configuring programmatically.
  4. Re-run and verify the service option gets appended automatically from --gbek.

Example fix

// before
options.view_as(GoogleCloudOptions).dataflow_service_options = ['use_gbek']
// after
options.view_as(SetupOptions).gbek = 'projects/p/secrets/gbek-key'
# use_gbek is added to dataflow_service_options automatically
Defensive patterns

Strategy: validation

Validate before calling

dso = options.view_as(GoogleCloudOptions).dataflow_service_options or []
assert 'use_gbek' not in dso or options.view_as(SetupOptions).gbek, 'set --gbek instead of use_gbek'

Try / catch

try:
    pipeline.run()
except ValueError as e:
    if 'use_gbek' in str(e):
        opts.view_as(GoogleCloudOptions).dataflow_service_options = [o for o in (opts.view_as(GoogleCloudOptions).dataflow_service_options or []) if o != 'use_gbek']
        opts.view_as(SetupOptions).gbek = 'gcp:secret:gbek-key'
        pipeline.run()

Prevention

When it happens

Trigger: Passing --dataflow_service_options=use_gbek (or appending 'use_gbek' to GoogleCloudOptions.dataflow_service_options) without also setting the --gbek SetupOption; detected when 'use_gbek' is present in dataflow_service_options while options.view_as(SetupOptions).gbek is unset.

Common situations: Misreading documentation and enabling GBEK via service options instead of the --gbek flag; leftover service options from a config where --gbek was later removed.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c10cd223f64d7946. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/dataflow/dataflow_runner.py:624


def _check_and_add_missing_options(options):
  # Type: (PipelineOptions) -> None

  """Validates and adds missing pipeline options depending on options set.

  :param options: PipelineOptions for this pipeline.
  """
  debug_options = options.view_as(DebugOptions)
  dataflow_service_options = options.view_as(
      GoogleCloudOptions).dataflow_service_options or []

  # Add use_gbek to dataflow_service_options if gbek is set.
  if options.view_as(SetupOptions).gbek:
    if 'use_gbek' not in dataflow_service_options:
      dataflow_service_options.append('use_gbek')
  elif 'use_gbek' in dataflow_service_options:
    raise ValueError(
        'Do not set use_gbek directly, pass in the --gbek pipeline option '
        'with a valid secret instead.')

  _add_runner_v2_missing_options(options)

  # Ensure that prime is specified as an experiment if specified as a dataflow
  # service option
  if 'enable_prime' in dataflow_service_options:
    debug_options.add_experiment('enable_prime')
  elif debug_options.lookup_experiment('enable_prime'):
    dataflow_service_options.append('enable_prime')

  options.view_as(
      GoogleCloudOptions).dataflow_service_options = dataflow_service_options

  sdk_location = options.view_as(SetupOptions).sdk_location
  if 'dev' in beam.version.__version__ and sdk_location == 'default':
    raise ValueError(

View on GitHub (pinned to 12126d8942)