apache/beam · error · ImportError

Pipeline option pickle_library=dill is set, but dill is not…

Error message

Pipeline option pickle_library=dill is set, but dill is not installed. Install apache-beam with the dill extras package e.g. apache-beam[dill].

What it means

apache_beam.internal.pickler.set_library selects the pickle backend (pickle, dill, cloudpickle) used by Beam to serialize callables/DAG fragments. If the pipeline option pickle_library=dill is requested but the dill package is not importable in the current environment, set_library raises ImportError pointing to the apache-beam[dill] extra. This happens at pipeline-construction/submission time, not during user code execution.

Solutions

Install the extra: pip install 'apache-beam[dill]'
Or run: pip install dill in both the job-submission and runtime (worker) environments
Remove --pickle_library=dill to use the default pickle backend
Pin dill in your worker setup/requirements files so Dataflow workers get it

Example fix

// before
pip install apache-beam
--pickle_library=dill
// after
pip install 'apache-beam[dill]'
--pickle_library=dill

Defensive patterns

Strategy: validation

Validate before calling

import importlib.util
dill_available = importlib.util.find_spec('dill') is not None
assert dill_available, "pip install 'apache-beam[dill]'"

Try / catch

try:
    pickler.set_library('dill')
except ImportError:
    pickler.set_library('default')  # fallback to stdlib pickle

Prevention

Install with extras: pip install 'apache-beam[dill]'
Keep dill in requirements/worker setup files for Dataflow
Verify `python -c 'import dill'` succeeds in submission and worker images

When it happens

Trigger: Launching a pipeline with --pickle_library=dill (PipelineOption pickle_library='dill') in an environment where `import dill` failed (dill not installed).

Common situations: Submitting to Google Cloud Dataflow where the submission environment has dill but workers do not, or vice versa; installing plain apache-beam without extras; using a requirements.txt that drops the dill dependency.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/81843f4d79fbe445. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/internal/pickler.py:110


def is_currently_dill():
  return desired_pickle_lib == dill_pickler


def is_currently_cloudpickle():
  return desired_pickle_lib == cloudpickle_pickler


def set_library(selected_library=DEFAULT_PICKLE_LIB):
  """ Sets pickle library that will be used. """
  global desired_pickle_lib

  if selected_library == 'default':
    selected_library = DEFAULT_PICKLE_LIB

  if selected_library == USE_DILL and not dill_pickler:
    raise ImportError(
        "Pipeline option pickle_library=dill is set, but dill is not "
        "installed. Install apache-beam with the dill extras package "
        "e.g. apache-beam[dill].")
  if selected_library == USE_DILL_UNSAFE and not dill_pickler:
    raise ImportError(
        "Pipeline option pickle_library=dill_unsafe is set, but dill is not "
        "installed. Install dill in job submission and runtime environments.")

  dill_is_requested = (
      selected_library == USE_DILL or selected_library == USE_DILL_UNSAFE)

  # If switching to or from dill, update the pickler hook overrides.
  if is_currently_dill() != dill_is_requested:
    dill_pickler.override_pickler_hooks(selected_library == USE_DILL)

  if dill_is_requested:
    desired_pickle_lib = dill_pickler
  elif selected_library == USE_CLOUDPICKLE:

View on GitHub (pinned to 12126d8942)