apache/beam · error · ValueError

Unexpected pipeline runner

Error message

Unexpected pipeline runner: %s. Valid values are %s or the fully qualified name of a PipelineRunner subclass.

What it means

create_runner() only accepts known runner names (StandardOptions.KNOWN_RUNNER_NAMES) or a fully qualified PipelineRunner subclass name. Any other string reaches the else branch and raises this ValueError listing the valid values.

Solutions

Use one of the valid names printed in the error (e.g. DirectRunner, DataflowRunner, FlinkRunner, SparkRunner)
Pass the fully qualified class name of a custom PipelineRunner subclass, e.g. 'mypkg.runners.MyRunner'
Check for typos and case in the runner name coming from flags/config

Example fix

// before
runner = create_runner('DirectRuner')
// after
runner = create_runner('DirectRunner')

Defensive patterns

Strategy: validation

Validate before calling

from apache_beam.options.pipeline_options import StandardOptions
VALID = set(StandardOptions.KNOWN_RUNNER_NAMES)
assert runner_name in VALID or '.' in runner_name, f'unknown runner: {runner_name}'

Try / catch

try:
    runner = create_runner(name)
except ValueError as e:
    print(e)  # lists valid values
    runner = create_runner('DirectRunner')

Prevention

Copy runner names from StandardOptions.KNOWN_RUNNER_NAMES instead of typing them
Keep runner name in one config constant
For custom runners, pass the fully qualified class name string

When it happens

Trigger: Calling create_runner() with an unrecognized runner_name string, e.g. create_runner('myrunner') or a typo like 'DirectRuner'; no ImportError occurred, the name simply failed to match known runners or a resolvable qualified class path.

Common situations: Typo in runner name from CLI flags or config files; passing a class object instead of its fully qualified string; older/newer Beam versions where the runner name is not in KNOWN_RUNNER_NAMES.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/74b23913e616cbbd. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/runner.py:97

      _RUNNER_MAP.get(runner_name.lower() + 'runner', runner_name))

  if '.' in runner_name:
    module, runner = runner_name.rsplit('.', 1)
    try:
      return getattr(importlib.import_module(module), runner)()
    except ImportError:
      if 'dataflow' in runner_name.lower():
        raise ImportError(
            'Google Cloud Dataflow runner not available, '
            'please install apache_beam[gcp]')
      elif 'interactive' in runner_name.lower():
        raise ImportError(
            'Interactive runner not available, '
            'please install apache_beam[interactive]')
      else:
        raise
  else:
    raise ValueError(
        'Unexpected pipeline runner: %s. Valid values are %s '
        'or the fully qualified name of a PipelineRunner subclass.' %
        (runner_name, ', '.join(StandardOptions.KNOWN_RUNNER_NAMES)))


class PipelineRunner(object):
  """A runner of a pipeline object.

  The base runner provides a run() method for visiting every node in the
  pipeline's DAG and executing the transforms computing the PValue in the node.

  A custom runner will typically provide implementations for some of the
  transform methods (ParDo, GroupByKey, Create, etc.). It may also
  provide a new implementation for clear_pvalue(), which is used to wipe out
  materialized values in order to reduce footprint.
  """
  def run(
      self,

View on GitHub (pinned to 12126d8942)