apache/beam · error · RuntimeError

This pipeline contains a DillCoder which requires the dill p

Error message

This pipeline contains a DillCoder which requires the dill package. Install the dill package with the dill extra e.g. apache-beam[dill]

What it means

DillCoder serializes objects with the optional 'dill' package. If dill is not importable in the environment, __init__ raises RuntimeError immediately so the pipeline fails fast with an actionable install hint instead of failing later during serialization.

Source

Thrown at sdks/python/apache_beam/coders/coders.py:927

  """Coder using Python's pickle functionality."""
  def _create_impl(self):
    dumps = pickle.dumps
    protocol = pickle.HIGHEST_PROTOCOL
    return coder_impl.CallbackCoderImpl(
        lambda x: dumps(x, protocol), pickle.loads)

  def as_deterministic_coder(self, step_label, error_message=None):
    return FastPrimitivesCoder(self, requires_deterministic=step_label)

  def to_type_hint(self):
    return Any


class DillCoder(_PickleCoderBase):
  """Coder using dill's pickle functionality."""
  def __init__(self):
    if not dill:
      raise RuntimeError(
          "This pipeline contains a DillCoder which requires "
          "the dill package. Install the dill package with the dill extra "
          "e.g. apache-beam[dill]")

  def _create_impl(self):
    return coder_impl.CallbackCoderImpl(maybe_dill_dumps, maybe_dill_loads)


class CloudpickleCoder(_PickleCoderBase):
  """Coder using Apache Beam's vendored Cloudpickle pickler."""
  def _create_impl(self):
    return coder_impl.CallbackCoderImpl(
        cloudpickle_pickler.dumps, cloudpickle_pickler.loads)


class DeterministicFastPrimitivesCoderV2(FastCoder):
  """Throws runtime errors when encoding non-deterministic values."""
  def __init__(self, coder, step_label):

View on GitHub (pinned to 12126d8942)

Solutions

Install dill: pip install apache-beam[dill] (or pip install dill)
Pin dill in requirements.txt / setup.py so the worker environment includes it
Avoid constructs that require DillCoder (e.g. dill-based pickling of lambdas) if dill cannot be installed

Example fix

// before
pip install apache-beam
// after
pip install 'apache-beam[dill]'

Defensive patterns

Strategy: validation

Validate before calling

import importlib.util
if importlib.util.find_spec('dill') is None:
    raise SystemExit("Install the dill package: pip install 'apache-beam[dill]'")

Type guard

def dill_available() -> bool:
    import importlib.util
    return importlib.util.find_spec('dill') is not None

Try / catch

try:
    run_pipeline(pipeline_options)
except RuntimeError as e:
    if 'DillCoder' in str(e):
        install_hint('pip install apache-beam[dill]')
    raise

Prevention

Add the [dill] extra to apache-beam installs in all environments
Pin dill in requirements.txt so workers get it
Smoke-test the pipeline locally before submitting to a runner

When it happens

Trigger: Using a transform (e.g. DillPickler-based lambdas/legacy pickling paths) that yields a DillCoder while the 'dill' package is not installed in the Python environment running the pipeline.

Common situations: Deploying a Beam pipeline to Dataflow or another runner where the base environment lacks dill; forgetting the [dill] extra when installing apache-beam; using pinned/dry environments for CI.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3be9637fc08aa217. Report an issue: GitHub.