apache/beam · error · ImportError

Google Cloud Dataflow runner not available, please install…

Error message

Google Cloud Dataflow runner not available, please install apache_beam[gcp]

What it means

PipelineOptions.create_runner resolves a runner by dotted module path; when the import fails with ImportError and the name mentions 'dataflow', it re-raises a clearer ImportError explaining that the Dataflow runner requires the apache_beam[gcp] extra. This translates a raw module-import failure into an actionable install hint.

Solutions

1. pip install apache_beam[gcp] before running with DataflowRunner
2. Verify the runner class name is spelled correctly (DataflowRunner)
3. Check that importlib can find the module: python -c 'from apache_beam.runners.dataflow import DataflowRunner'

Example fix

# before
pip install apache-beam
--runner=DataflowRunner  # ImportError: not available
# after
pip install 'apache-beam[gcp]'
--runner=DataflowRunner

import importlib.util
if runner_name.endswith('DataflowRunner') and importlib.util.find_spec('google.cloud.storage') is None:
    raise SystemExit("Install GCP extras first: pip install 'apache_beam[gcp]'")

def runner_importable(runner_name: str) -> bool:
    import importlib
    if '.' not in runner_name:
        return True
    module, runner = runner_name.rsplit('.', 1)
    try:
        importlib.import_module(module)
        return True
    except ImportError:
        return False

try:
    runner = PipelineOptions(argv).runner  # or create_runner(runner_name)
except ImportError as e:
    if 'apache_beam[gcp]' in str(e):
        subprocess.run(['pip', 'install', 'apache_beam[gcp]'], check=True)
        runner = create_runner(runner_name)
    else:
        raise

When it happens

Trigger: Passing --runner=DataflowRunner (or --runner=apache_beam.runners.dataflow.DataflowRunner) without google-cloud dependencies installed, so importing the dataflow runner module raises ImportError.

Common situations: Local venvs with only apache-beam base install; CI jobs that submit to Dataflow without apache_beam[gcp]; typo'd runner class names alongside genuinely missing GCP extras.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

• Can not query metrics. Job id is unknown.
• Coder for the GroupByKey operation
• CombineFn.setup and CombineFn.teardown are not supported…
• Could not find element
• Could not translate the internal step name %r.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6dbe7e9b4de0d823. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/runner.py:87

  Raises:
    RuntimeError: if an invalid runner name is used.
  """

  # Get the qualified runner name by using the lower case runner name. If that
  # fails try appending the name with 'runner' and check if it matches.
  # If that also fails, use the given runner name as is.
  runner_name = _RUNNER_MAP.get(
      runner_name.lower(),
      _RUNNER_MAP.get(runner_name.lower() + 'runner', runner_name))

  if '.' in runner_name:
    module, runner = runner_name.rsplit('.', 1)
    try:
      return getattr(importlib.import_module(module), runner)()
    except ImportError:
      if 'dataflow' in runner_name.lower():
        raise ImportError(
            'Google Cloud Dataflow runner not available, '
            'please install apache_beam[gcp]')
      elif 'interactive' in runner_name.lower():
        raise ImportError(
            'Interactive runner not available, '
            'please install apache_beam[interactive]')
      else:
        raise
  else:
    raise ValueError(
        'Unexpected pipeline runner: %s. Valid values are %s '
        'or the fully qualified name of a PipelineRunner subclass.' %
        (runner_name, ', '.join(StandardOptions.KNOWN_RUNNER_NAMES)))


class PipelineRunner(object):
  """A runner of a pipeline object.

View on GitHub (pinned to 12126d8942)