{"record":{"id":"81843f4d79fbe445","repo":"apache/beam","slug":"pipeline-option-pickle-library-dill-is-set-but-dill-is-not","errorCode":null,"errorMessage":"Pipeline option pickle_library=dill is set, but dill is not installed. Install apache-beam with the dill extras package e.g. apache-beam[dill].","messagePattern":"Pipeline option pickle_library=dill is set, but dill is not installed\\. Install apache-beam with the dill extras package e\\.g\\. apache-beam\\[dill\\]\\.","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/internal/pickler.py","lineNumber":110,"sourceCode":"\n\ndef is_currently_dill():\n  return desired_pickle_lib == dill_pickler\n\n\ndef is_currently_cloudpickle():\n  return desired_pickle_lib == cloudpickle_pickler\n\n\ndef set_library(selected_library=DEFAULT_PICKLE_LIB):\n  \"\"\" Sets pickle library that will be used. \"\"\"\n  global desired_pickle_lib\n\n  if selected_library == 'default':\n    selected_library = DEFAULT_PICKLE_LIB\n\n  if selected_library == USE_DILL and not dill_pickler:\n    raise ImportError(\n        \"Pipeline option pickle_library=dill is set, but dill is not \"\n        \"installed. Install apache-beam with the dill extras package \"\n        \"e.g. apache-beam[dill].\")\n  if selected_library == USE_DILL_UNSAFE and not dill_pickler:\n    raise ImportError(\n        \"Pipeline option pickle_library=dill_unsafe is set, but dill is not \"\n        \"installed. Install dill in job submission and runtime environments.\")\n\n  dill_is_requested = (\n      selected_library == USE_DILL or selected_library == USE_DILL_UNSAFE)\n\n  # If switching to or from dill, update the pickler hook overrides.\n  if is_currently_dill() != dill_is_requested:\n    dill_pickler.override_pickler_hooks(selected_library == USE_DILL)\n\n  if dill_is_requested:\n    desired_pickle_lib = dill_pickler\n  elif selected_library == USE_CLOUDPICKLE:","sourceCodeStart":92,"sourceCodeEnd":128,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/internal/pickler.py#L92-L128","documentation":"apache_beam.internal.pickler.set_library selects the pickle backend (pickle, dill, cloudpickle) used by Beam to serialize callables/DAG fragments. If the pipeline option pickle_library=dill is requested but the dill package is not importable in the current environment, set_library raises ImportError pointing to the apache-beam[dill] extra. This happens at pipeline-construction/submission time, not during user code execution.","triggerScenarios":"Launching a pipeline with --pickle_library=dill (PipelineOption pickle_library='dill') in an environment where `import dill` failed (dill not installed).","commonSituations":"Submitting to Google Cloud Dataflow where the submission environment has dill but workers do not, or vice versa; installing plain apache-beam without extras; using a requirements.txt that drops the dill dependency.","solutions":["Install the extra: pip install 'apache-beam[dill]'","Or run: pip install dill in both the job-submission and runtime (worker) environments","Remove --pickle_library=dill to use the default pickle backend","Pin dill in your worker setup/requirements files so Dataflow workers get it"],"exampleFix":"// before\npip install apache-beam\n--pickle_library=dill\n// after\npip install 'apache-beam[dill]'\n--pickle_library=dill","handlingStrategy":"validation","validationCode":"import importlib.util\ndill_available = importlib.util.find_spec('dill') is not None\nassert dill_available, \"pip install 'apache-beam[dill]'\"","typeGuard":null,"tryCatchPattern":"try:\n    pickler.set_library('dill')\nexcept ImportError:\n    pickler.set_library('default')  # fallback to stdlib pickle","preventionTips":["Install with extras: pip install 'apache-beam[dill]'","Keep dill in requirements/worker setup files for Dataflow","Verify `python -c 'import dill'` succeeds in submission and worker images"],"tags":["python","dependency","pickle","dill","installation"],"backgroundTag":"missing-optional-dependency","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}