apache/beam · error · ImportError

Google Cloud Dataflow runner not available, please install a

Error message

Google Cloud Dataflow runner not available, please install apache_beam[gcp]

What it means

Identical in spirit to error 3400 but raised in DataflowRunner.run_pipeline: submitting a job to Google Cloud Dataflow requires the apiclient module, whose Google Cloud dependencies ship only with the 'apache-beam[gcp]' extra. The import is deferred to job submission so local runs never require those packages.

Source

Thrown at sdks/python/apache_beam/runners/dataflow/dataflow_runner.py:373

          'Disabling Dataflow Portable Runner no longer supported '
          'using Beam Python %s.' % beam.version.__version__)

    # Label goog-dataflow-notebook if job is started from notebook.
    if is_in_notebook():
      notebook_version = (
          'goog-dataflow-notebook=' +
          beam.version.__version__.replace('.', '_'))
      if options.view_as(GoogleCloudOptions).labels:
        options.view_as(GoogleCloudOptions).labels.append(notebook_version)
      else:
        options.view_as(GoogleCloudOptions).labels = [notebook_version]

    # Import here to avoid adding the dependency for local running scenarios.
    try:
      # pylint: disable=wrong-import-order, wrong-import-position
      from apache_beam.runners.dataflow.internal import apiclient
    except ImportError:
      raise ImportError(
          'Google Cloud Dataflow runner not available, '
          'please install apache_beam[gcp]')

    _check_and_add_missing_options(options)

    # Convert all side inputs into a form acceptable to Dataflow.
    if pipeline:
      pipeline.visit(self.combinefn_visitor())

      pipeline.visit(
          self.side_input_visitor(
              deterministic_key_coders=not options.view_as(
                  TypeOptions).allow_non_deterministic_key_coders))

      # Performing configured PTransform overrides. Note that this is currently
      # done before Runner API serialization, since the new proto needs to
      # contain any added PTransforms.
      pipeline.replace_all(DataflowRunner._PTRANSFORM_OVERRIDES)

View on GitHub (pinned to 12126d8942)

Solutions

pip install 'apache-beam[gcp]' in the submission environment.
Verify the import: python -c "from apache_beam.runners.dataflow.internal import apiclient".
If using a constrained image, add the GCP extras to requirements: apache-beam[gcp]==<your-version>.
Use --requirements_file / --save_main_session so worker environments include the same extras.

Example fix

// before
pip install apache-beam
with beam.Pipeline(runner='DataflowRunner', options=options) as p: ...
// after
pip install 'apache-beam[gcp]'
with beam.Pipeline(runner='DataflowRunner', options=options) as p: ...

Defensive patterns

Strategy: fallback

Validate before calling

try:
    from apache_beam.runners.dataflow.internal import apiclient
    CAN_SUBMIT_DATAFLOW = True
except ImportError:
    CAN_SUBMIT_DATAFLOW = False

Try / catch

try:
    result = pipeline.run()
except ImportError:
    result = pipeline_local.run()  # DirectRunner fallback

Prevention

Install 'apache-beam[gcp]' in every environment that submits Dataflow jobs
Include the [gcp] extra in Docker images used for submission
Pre-flight import check in the submission script

When it happens

Trigger: Calling pipeline.run() with DataflowRunner when google-api-client / related GCP packages are missing, so 'from apache_beam.runners.dataflow.internal import apiclient' raises ImportError.

Common situations: Installing plain apache-beam in CI or a container before submitting to Dataflow; deployment images that trimmed the [gcp] extras; switching runners from DirectRunner to DataflowRunner at submit time.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/41b1c3e9c5a492c2. Report an issue: GitHub.