apache/beam · error · ImportError

Google Cloud Dataflow runner not available, please install a

Error message

Google Cloud Dataflow runner not available, please install apache_beam[gcp]

What it means

This ImportError is raised by DataflowMetrics (and the Dataflow runner generally) when the Dataflow-specific client libraries cannot be imported. The 'apache_beam.runners.dataflow.internal.apiclient' module depends on extra Google Cloud packages (google-api-client, google-cloud-dataflow-client, etc.) that are only installed with the 'gcp' extra of apache-beam. The library raises it lazily so local, non-GCP usage never needs the dependency.

Source

Thrown at sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py:281

    }


def main(argv):
  """Print the metric results for the dataflow --job_id and --project.

  Instead of running an entire pipeline which takes several minutes, use this
  main method to display MetricResults for a specific --job_id and --project
  which takes only a few seconds.
  """
  # TODO(https://github.com/apache/beam/issues/19452): The MetricResults do not
  # show translated step names as the job_graph is not provided to
  # DataflowMetrics. Import here to avoid adding the dependency for local
  # running scenarios.
  try:
    # pylint: disable=wrong-import-order, wrong-import-position
    from apache_beam.runners.dataflow.internal import apiclient
  except ImportError:
    raise ImportError(
        'Google Cloud Dataflow runner not available, '
        'please install apache_beam[gcp]')
  if argv[0] == __file__:
    argv = argv[1:]
  parser = argparse.ArgumentParser()
  parser.add_argument(
      '-j', '--job_id', type=str, help='The job id to query metrics for.')
  parser.add_argument(
      '-p',
      '--project',
      type=str,
      help='The project name to query metrics for.')
  flags = parser.parse_args(argv)

  # Get a Dataflow API client and set its project and job_id in the options.
  options = PipelineOptions()
  gcloud_options = options.view_as(GoogleCloudOptions)
  gcloud_options.project = flags.project

View on GitHub (pinned to 12126d8942)

Solutions

  1. Install the GCP extras: pip install 'apache-beam[gcp]'
  2. Verify the internal import works: python -c "from apache_beam.runners.dataflow.internal import apiclient"
  3. If dependency conflicts block the upgrade, install the minimal GCP packages (google-api-client, google-cloud-dataflow-client) into the environment.
  4. If you do not actually need Dataflow, use a local runner (DirectRunner) instead of DataflowMetrics.

Example fix

// before
pip install apache-beam
results = pipeline.result.metrics()
// after
pip install 'apache-beam[gcp]'
results = pipeline.result.metrics()
Defensive patterns

Strategy: fallback

Validate before calling

try:
    from apache_beam.runners.dataflow.internal import apiclient
except ImportError:
    raise SystemExit("Install GCP extras: pip install 'apache-beam[gcp]'")

Try / catch

try:
    results = pipeline.result.metrics()
except ImportError:
    results = None  # metrics unavailable without [gcp] extras

Prevention

When it happens

Trigger: Instantiating DataflowMetrics (e.g. results = pipeline.result.metrics()) or otherwise touching the Dataflow metrics API when apache-beam was installed without the [gcp] extra, so the apiclient import fails.

Common situations: Installing with 'pip install apache-beam' instead of 'pip install apache-beam[gcp]'; slim/venv deployments with Google Cloud extras stripped; running a Dataflow metrics query from a local environment with only core Beam installed.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/88a0c6132acd8fdb. Report an issue: GitHub.