apache/beam · error · ImportError

Pipeline option pickle_library=dill_unsafe is set, but dill…

Error message

Pipeline option pickle_library=dill_unsafe is set, but dill is not installed. Install dill in job submission and runtime environments.

What it means

The same set_library backend selector in apache_beam/internal/pickler.py, for the dill_unsafe variant. Requesting pickle_library=dill_unsafe without the dill package installed raises ImportError instructing the user to install dill in both the job submission and runtime environments (the unsafe variant skips some safety checks and needs dill present at both ends).

Solutions

1. pip install dill in the job submission environment AND in the runtime/worker environment
2. Use the official extra: pip install 'apache-beam[dill]'
3. Switch to pickle_library=default (stdlib pickle) if dill cannot be installed
4. Bake dill into your custom worker container image

Example fix

// before
--pickle_library=dill_unsafe  # ImportError: dill not installed
// after
pip install dill  # or: pip install 'apache-beam[dill]'
--pickle_library=dill_unsafe

import importlib.util
assert importlib.util.find_spec('dill') is not None, 'dill required for pickle_library=dill_unsafe'

try:
    pickler.set_library('dill_unsafe')
except ImportError:
    pickler.set_library('default')

When it happens

Trigger: Running with pipeline option pickle_library=dill_unsafe while dill is not importable in the current environment.

Common situations: Teams choosing dill_unsafe for lambda/nested-function serialization but forgetting the dependency on Dataflow workers; container images built without dill; version skew where dill exists locally but not in the job service image.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

• Pipeline option pickle_library=dill is set, but dill is not…
• can't (safely) pickle generator objects
• Cannot find default Beam SDK tar file
• Cannot pickle closed files
• Cannot pickle file as it cannot be read

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/dbdd323f99f8fc1f. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/internal/pickler.py:115

def is_currently_cloudpickle():
  return desired_pickle_lib == cloudpickle_pickler


def set_library(selected_library=DEFAULT_PICKLE_LIB):
  """ Sets pickle library that will be used. """
  global desired_pickle_lib

  if selected_library == 'default':
    selected_library = DEFAULT_PICKLE_LIB

  if selected_library == USE_DILL and not dill_pickler:
    raise ImportError(
        "Pipeline option pickle_library=dill is set, but dill is not "
        "installed. Install apache-beam with the dill extras package "
        "e.g. apache-beam[dill].")
  if selected_library == USE_DILL_UNSAFE and not dill_pickler:
    raise ImportError(
        "Pipeline option pickle_library=dill_unsafe is set, but dill is not "
        "installed. Install dill in job submission and runtime environments.")

  dill_is_requested = (
      selected_library == USE_DILL or selected_library == USE_DILL_UNSAFE)

  # If switching to or from dill, update the pickler hook overrides.
  if is_currently_dill() != dill_is_requested:
    dill_pickler.override_pickler_hooks(selected_library == USE_DILL)

  if dill_is_requested:
    desired_pickle_lib = dill_pickler
  elif selected_library == USE_CLOUDPICKLE:
    desired_pickle_lib = cloudpickle_pickler
  else:
    raise ValueError(f'Unknown pickler library: {selected_library}')

View on GitHub (pinned to 12126d8942)