apache/beam · error · ImportError

DaskRunner is not available. Please install apache_beam[dask

Error message

DaskRunner is not available. Please install apache_beam[dask].

What it means

DaskRunner.run_pipeline could not import dask.distributed, meaning the Dask scheduler extra is not installed in this environment; the runner cannot launch workers and fails fast with install instructions.

Source

Thrown at sdks/python/apache_beam/runners/dask/dask_runner.py:235

        self.bags[transform_node] = op.apply(**op_kws)

    return DaskBagVisitor()

  @staticmethod
  def is_fnapi_compatible():
    return False

  def run_pipeline(self, pipeline, options):
    import dask

    # TODO(alxmrs): Create interactive notebook support.
    if is_in_notebook():
      raise NotImplementedError('interactive support will come later!')

    try:
      import dask.distributed as ddist
    except ImportError:
      raise ImportError(
          'DaskRunner is not available. Please install apache_beam[dask].')

    dask_options = options.view_as(DaskOptions).get_all_options(
        drop_default=True, current_only=True)
    bag_kwargs = DaskOptions._extract_bag_kwargs(dask_options)
    client = ddist.Client(**dask_options)

    pipeline.replace_all(dask_overrides())

    dask_visitor = self.to_dask_bag_visitor(bag_kwargs)
    pipeline.visit(dask_visitor)
    # The dictionary in this visitor keeps a mapping of every Beam
    # PTransform to the equivalent Bag operation. This is highly
    # redundant. Thus, we can get away with computing just the last
    # value, which should be connected to the full Bag Task Graph.
    opt_graph = dask.optimize(list(dask_visitor.bags.values())[-1])
    futures = client.compute(opt_graph)
    return DaskRunnerResult(client, futures)

View on GitHub (pinned to 12126d8942)

Solutions

1. Install the extra: `pip install apache-beam[dask]`.
2. Or install dask and distributed separately: `pip install dask distributed`.
3. Pin compatible versions of dask/distributed if apache-beam's extra resolves a too-new version.

Example fix

// before
pip install apache-beam
// after
pip install "apache-beam[dask]"

try:
    import dask.distributed  # noqa
except ImportError:
    raise SystemExit('Install apache_beam[dask] to use DaskRunner')

def dask_available() -> bool:
    try:
        import dask.distributed  # noqa: F401
        return True
    except ImportError:
        return False

try:
    with beam.Pipeline(runner='DaskRunner', options=opts) as p:
        ...
except ImportError as e:
    logging.error('%s; run: pip install apache-beam[dask]', e)

When it happens

Trigger: Running a pipeline with DaskRunner in an environment where `import dask.distributed` fails (dask/distributed not installed).

Common situations: CI or production environments where only the base apache-beam package was pip-installed, without the [dask] extra.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

• Failed to import hdfs. You can ensure it is installed by ins
• Module {parts[0]} not found in sys.modules
• google-cloud-bigquery-storage is required for ReadBigQueryCh
• pyarrow is required for ReadBigQueryChangeHistory. Install i
• Could not import joblib in this execution environment. For h

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4dda2d1132ace0b4. Report an issue: GitHub.