apache/beam · error · ValueError

Unknown or inapplicable phase for pre_optimize

Error message

Unknown or inapplicable phase for pre_optimize: %s

What it means

The --pre_optimize option lets users run pipeline-optimization translations before submitting to Dataflow, but only a whitelisted set of phases is applicable on this runner path. Passing any other phase name raises this ValueError listing the offending phase.

Solutions

Change pre_optimize to only use supported phases, e.g. --pre_optimize=pack_combiners.
Remove the pre_optimize option entirely to use the default optimization behavior.
Check apache_beam/runners/transform.py translations for phases supported by your Beam version.
Upgrade apache-beam if a newer version whitelists the phase you need.

Example fix

// before
--pre_optimize=all
// after
--pre_optimize=pack_combiners

Defensive patterns

Strategy: validation

Validate before calling

SUPPORTED = {'pack_combiners'}
phases = [p for p in opts.pre_optimize.split(',') if p]
assert all(p in SUPPORTED for p in phases), f'unsupported pre_optimize: {phases}'

Try / catch

try:
    pipeline.run()
except ValueError as e:
    if 'pre_optimize' in str(e):
        opts.view_as(SetupOptions).pre_optimize = 'pack_combiners'
        pipeline.run()

Prevention

Only pass --pre_optimize=pack_combiners on the Dataflow non-portable path
Validate pre_optimize values in wrapper scripts before submit
Drop pre_optimize unless you specifically need packed combiners

When it happens

Trigger: Setting pipeline option pre_optimize to a comma-separated list containing anything other than 'pack_combiners' (e.g. pre_optimize=all or pre_optimize=sort_stages) when using the Dataflow non-portable path.

Common situations: Copying --pre_optimize flags from documentation for a different runner; assuming all translations.* phases are valid pre_optimize phases.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b33bf2e32d096b82. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/dataflow/dataflow_runner.py:480

    # Optimize the pipeline if it not streaming and the pre_optimize
    # experiment is set.
    if not options.view_as(StandardOptions).streaming:
      pre_optimize = options.view_as(DebugOptions).lookup_experiment(
          'pre_optimize', 'default').lower()
      from apache_beam.runners.portability.fn_api_runner import translations
      if pre_optimize == 'none':
        phases = []
      elif pre_optimize == 'default' or pre_optimize == 'all':
        phases = [translations.pack_combiners, translations.sort_stages]
      else:
        phases = []
        for phase_name in pre_optimize.split(','):
          # For now, these are all we allow.
          if phase_name in ('pack_combiners', ):
            phases.append(getattr(translations, phase_name))
          else:
            raise ValueError(
                'Unknown or inapplicable phase for pre_optimize: %s' %
                phase_name)
        phases.append(translations.sort_stages)

      if phases:
        self.proto_pipeline = translations.optimize_pipeline(
            self.proto_pipeline,
            phases=phases,
            known_runner_urns=frozenset(),
            partial=True)

    # Add setup_options for all the BeamPlugin imports
    setup_options = options.view_as(SetupOptions)
    plugins = BeamPlugin.get_all_plugin_paths()
    if setup_options.beam_plugins is not None:
      plugins = list(set(plugins + setup_options.beam_plugins))
    setup_options.beam_plugins = plugins

View on GitHub (pinned to 12126d8942)