apache/beam · error · RuntimeError

This pipeline runs with the pipeline option…

Error message

This pipeline runs with the pipeline option --update_compatibility_version=2.67.0 or earlier. When running with this option on SDKs 2.68.0 or later, you must ensure dill==0.3.1.1 is installed.. Found dill version '{dill.__version__}

What it means

Same _verify_dill_compat guard as the missing-dill case: with --update_compatibility_version=2.67.0 or earlier on SDKs 2.68.0+, Beam requires dill==0.3.1.1, and raises RuntimeError if a different dill version is found. (The rendered message interpolates the actual installed version into the f-string placeholder.)

Solutions

  1. Pin dill to exactly 0.3.1.1: `pip install dill==0.3.1.1` (check with `pip show dill`).
  2. Add a constraints file or extra_deps flag (Beam's --extra_package/requirements) ensuring dill==0.3.1.1 on workers too.
  3. Raise --update_compatibility_version above 2.67.0 if legacy pickles are not required, avoiding the strict dill pin.

Example fix

// before
pip show dill  # Version: 0.3.8
// after
pip install dill==0.3.1.1
Defensive patterns

Strategy: validation

Validate before calling

import dill
if dill.__version__ != '0.3.1.1':
    raise SystemExit(f'Need dill==0.3.1.1, found {dill.__version__}')

Try / catch

try:
    pipeline.run().wait_until_finish()
except RuntimeError as e:
    if 'Found dill version' in str(e):
        subprocess.run(['pip', 'install', 'dill==0.3.1.1'], check=True)

Prevention

When it happens

Trigger: Running an update-compatibility pipeline (encode_type_2_67_0 / _unpickle_type_2_67_0 code path) where dill is installed but its __version__ is not '0.3.1.1' (e.g. dill 0.3.8, 0.3.1.0, or 0.4.x).

Common situations: pip resolving a newer dill as a transitive dependency; environment drift between the machine that wrote the pipeline and the runner; upgrading dill for another library and breaking Beam's update compatibility.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/fdd28e7c529dfc6c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/coders/coder_impl.py:372

DATACLASS_KW_ONLY_TYPE = 105

# Types that can be encoded as iterables, but are not literally
# lists, etc. due to being lazy.  The actual type is not preserved
# through encoding, only the elements. This is particularly useful
# for the value list types created in GroupByKey.
_ITERABLE_LIKE_TYPES = set()  # type: Set[Type]


def _verify_dill_compat():
  base_error = (
      "This pipeline runs with the pipeline option "
      "--update_compatibility_version=2.67.0 or earlier. "
      "When running with this option on SDKs 2.68.0 or "
      "later, you must ensure dill==0.3.1.1 is installed.")
  if not dill:
    raise RuntimeError(base_error + ". Dill is not installed.")
  if dill.__version__ != "0.3.1.1":
    raise RuntimeError(base_error + f". Found dill version '{dill.__version__}")


class FastPrimitivesCoderImpl(StreamCoderImpl):
  """For internal use only; no backwards-compatibility guarantees."""
  def __init__(
      self,
      fallback_coder_impl,
      requires_deterministic_step_label=None,
      force_use_dill=False,
      use_relative_filepaths=True):
    self.fallback_coder_impl = fallback_coder_impl
    self.iterable_coder_impl = IterableCoderImpl(self)
    self.requires_deterministic_step_label = requires_deterministic_step_label
    self.warn_deterministic_fallback = True
    self.force_use_dill = force_use_dill
    self.use_relative_filepaths = use_relative_filepaths

  @staticmethod

View on GitHub (pinned to 12126d8942)