apache/beam · error · RuntimeError

This pipeline runs with the pipeline option…

Error message

This pipeline runs with the pipeline option --update_compatibility_version=2.67.0 or earlier. When running with this option on SDKs 2.68.0 or later, you must ensure dill==0.3.1.1 is installed.. Dill is not installed.

What it means

When a pipeline uses --update_compatibility_version=2.67.0 or earlier, Beam 2.68.0+ requires dill==0.3.1.1 for cross-version pickling compatibility. _verify_dill_compat raises this RuntimeError if the dill module is not installed at all.

Solutions

1. Install the exact required version: `pip install dill==0.3.1.1`.
2. Rebuild/publish custom worker containers or dependency bundles including dill==0.3.1.1.
3. If legacy compatibility is not needed, drop --update_compatibility_version or set it to a newer version so the legacy code path is not used.

Example fix

// before
pip install apache-beam==2.68.0
// after
pip install apache-beam==2.68.0 dill==0.3.1.1

try:
    import dill
except ImportError:
    raise SystemExit('pip install dill==0.3.1.1 before running update-compatibility pipelines')

try:
    pipeline.run().wait_until_finish()
except RuntimeError as e:
    if 'Dill is not installed' in str(e):
        subprocess.run(['pip', 'install', 'dill==0.3.1.1'], check=True)

When it happens

Trigger: Running an update/migration pipeline with --update_compatibility_version<=2.67.0 on SDK 2.68.0+ in an environment where the dill package is absent, and encoding/unpickling a legacy-coded value (encode_type_2_67_0 / _unpickle_type_2_67_0).

Common situations: Slimmed runtime environments (Dataflow worker, custom containers, Flink/Spark bundles) where dill was stripped; manually curated dependency sets that dropped dill after upgrading Beam.

Understand the failure class

Background: "not installed", "pip install", "required for": how missing-dependency errors surface across open-source libraries — this error's family across 34 libraries.

Related errors

• can't (safely) pickle generator objects
• This pipeline runs with the pipeline option…
• cannot check importability of
• Could not find code object with path
• Could not find code object with path

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/500331b8f7ebd151. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/coders/coder_impl.py:370

ENUM_TYPE = 103
NESTED_STATE_TYPE = 104
DATACLASS_KW_ONLY_TYPE = 105

# Types that can be encoded as iterables, but are not literally
# lists, etc. due to being lazy.  The actual type is not preserved
# through encoding, only the elements. This is particularly useful
# for the value list types created in GroupByKey.
_ITERABLE_LIKE_TYPES = set()  # type: Set[Type]


def _verify_dill_compat():
  base_error = (
      "This pipeline runs with the pipeline option "
      "--update_compatibility_version=2.67.0 or earlier. "
      "When running with this option on SDKs 2.68.0 or "
      "later, you must ensure dill==0.3.1.1 is installed.")
  if not dill:
    raise RuntimeError(base_error + ". Dill is not installed.")
  if dill.__version__ != "0.3.1.1":
    raise RuntimeError(base_error + f". Found dill version '{dill.__version__}")


class FastPrimitivesCoderImpl(StreamCoderImpl):
  """For internal use only; no backwards-compatibility guarantees."""
  def __init__(
      self,
      fallback_coder_impl,
      requires_deterministic_step_label=None,
      force_use_dill=False,
      use_relative_filepaths=True):
    self.fallback_coder_impl = fallback_coder_impl
    self.iterable_coder_impl = IterableCoderImpl(self)
    self.requires_deterministic_step_label = requires_deterministic_step_label
    self.warn_deterministic_fallback = True
    self.force_use_dill = force_use_dill
    self.use_relative_filepaths = use_relative_filepaths

View on GitHub (pinned to 12126d8942)