apache/beam · error · ValueError

Pipeline has validations errors:

Error message

Pipeline has validations errors:

What it means

After constructing a runner, Pipeline.__init__ runs PipelineOptionsValidator which collects validation errors for the option set against the chosen runner. If any errors are found they are joined and raised as a single ValueError listing every problem.

Source

Thrown at sdks/python/apache_beam/pipeline.py:226

      raise TypeError(
          'Runner %s is not a PipelineRunner object or the '
          'name of a registered runner.' % runner)

    # Runner can override the default pickler to be used.
    if (self._options.view_as(SetupOptions).pickle_library == 'default' and
        runner.default_pickle_library_override()):
      logging.info(
          "Runner defaulting to pickling library: %s.",
          runner.default_pickle_library_override())
      self._options.view_as(
          SetupOptions).pickle_library = runner.default_pickle_library_override(
          )
    pickler.set_library(self._options.view_as(SetupOptions).pickle_library)

    # Validate pipeline options
    errors = PipelineOptionsValidator(self._options, runner).validate()
    if errors:
      raise ValueError(
          'Pipeline has validations errors: \n' + '\n'.join(errors))

    typecoders.registry.update_compatibility_version = self._options.view_as(
        StreamingOptions).update_compatibility_version

    # set default experiments for portable runners
    # (needs to occur prior to pipeline construction)
    if runner.is_fnapi_compatible():
      experiments = (self._options.view_as(DebugOptions).experiments or [])
      if not 'beam_fn_api' in experiments:
        experiments.append('beam_fn_api')
        self._options.view_as(DebugOptions).experiments = experiments

    self.local_tempdir = tempfile.mkdtemp(prefix='beam-pipeline-temp')

    # Default runner to be used.
    self.runner = runner

View on GitHub (pinned to 12126d8942)

Solutions

  1. Read the full error list in the message; each line names an invalid option.
  2. Fix or remove the flagged PipelineOptions entries.
  3. Supply required options for the runner (e.g. project, region, temp_location for Dataflow).
  4. Verify option flags/names against the runner's documented option class.
  5. Test options with PipelineOptionsValidator or by constructing the pipeline before submitting.

Example fix

// before
options = PipelineOptions(['--streaming', '--worker_machine_type=n1-standard-1'])
p = Pipeline(runner='DirectRunner', options=options)
// after
options = PipelineOptions(['--worker_machine_type=n1-standard-1'])
p = Pipeline(runner='DirectRunner', options=options)
Defensive patterns

Strategy: validation

Validate before calling

from apache_beam.options.pipeline_options_validator import PipelineOptionsValidator
errors = PipelineOptionsValidator(options, runner).validate()
assert not errors, '\n'.join(errors)

Try / catch

try:
    p = Pipeline(options=options)
except ValueError as e:
    print('Option validation failed:', e)
    # fix flagged options before retrying

Prevention

When it happens

Trigger: Constructing Pipeline(options=PipelineOptions([...])) with options invalid for the runner, e.g. streaming-specific flags with a non-streaming runner, missing required flags (like --project/--region for Dataflow), or flags only valid for a different runner.

Common situations: Running on Dataflow without required GCP options; passing --streaming to a batch runner; copying flags between runners where they are incompatible; typos in option names so validation sees unexpected/invalid values.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6225aeb4adfb8da9. Report an issue: GitHub.