{"record":{"id":"dbdd323f99f8fc1f","repo":"apache/beam","slug":"pipeline-option-pickle-library-dill-unsafe-is-set-but-dill","errorCode":null,"errorMessage":"Pipeline option pickle_library=dill_unsafe is set, but dill is not installed. Install dill in job submission and runtime environments.","messagePattern":"Pipeline option pickle_library=dill_unsafe is set, but dill is not installed\\. Install dill in job submission and runtime environments\\.","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/internal/pickler.py","lineNumber":115,"sourceCode":"\ndef is_currently_cloudpickle():\n  return desired_pickle_lib == cloudpickle_pickler\n\n\ndef set_library(selected_library=DEFAULT_PICKLE_LIB):\n  \"\"\" Sets pickle library that will be used. \"\"\"\n  global desired_pickle_lib\n\n  if selected_library == 'default':\n    selected_library = DEFAULT_PICKLE_LIB\n\n  if selected_library == USE_DILL and not dill_pickler:\n    raise ImportError(\n        \"Pipeline option pickle_library=dill is set, but dill is not \"\n        \"installed. Install apache-beam with the dill extras package \"\n        \"e.g. apache-beam[dill].\")\n  if selected_library == USE_DILL_UNSAFE and not dill_pickler:\n    raise ImportError(\n        \"Pipeline option pickle_library=dill_unsafe is set, but dill is not \"\n        \"installed. Install dill in job submission and runtime environments.\")\n\n  dill_is_requested = (\n      selected_library == USE_DILL or selected_library == USE_DILL_UNSAFE)\n\n  # If switching to or from dill, update the pickler hook overrides.\n  if is_currently_dill() != dill_is_requested:\n    dill_pickler.override_pickler_hooks(selected_library == USE_DILL)\n\n  if dill_is_requested:\n    desired_pickle_lib = dill_pickler\n  elif selected_library == USE_CLOUDPICKLE:\n    desired_pickle_lib = cloudpickle_pickler\n  else:\n    raise ValueError(f'Unknown pickler library: {selected_library}')\n","sourceCodeStart":97,"sourceCodeEnd":132,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/internal/pickler.py#L97-L132","documentation":"The same set_library backend selector in apache_beam/internal/pickler.py, for the dill_unsafe variant. Requesting pickle_library=dill_unsafe without the dill package installed raises ImportError instructing the user to install dill in both the job submission and runtime environments (the unsafe variant skips some safety checks and needs dill present at both ends).","triggerScenarios":"Running with pipeline option pickle_library=dill_unsafe while dill is not importable in the current environment.","commonSituations":"Teams choosing dill_unsafe for lambda/nested-function serialization but forgetting the dependency on Dataflow workers; container images built without dill; version skew where dill exists locally but not in the job service image.","solutions":["pip install dill in the job submission environment AND in the runtime/worker environment","Use the official extra: pip install 'apache-beam[dill]'","Switch to pickle_library=default (stdlib pickle) if dill cannot be installed","Bake dill into your custom worker container image"],"exampleFix":"// before\n--pickle_library=dill_unsafe  # ImportError: dill not installed\n// after\npip install dill  # or: pip install 'apache-beam[dill]'\n--pickle_library=dill_unsafe","handlingStrategy":"validation","validationCode":"import importlib.util\nassert importlib.util.find_spec('dill') is not None, 'dill required for pickle_library=dill_unsafe'","typeGuard":null,"tryCatchPattern":"try:\n    pickler.set_library('dill_unsafe')\nexcept ImportError:\n    pickler.set_library('default')","preventionTips":["Install dill in BOTH job-submission and runtime environments","Bake dill into custom worker container images","Prefer 'default' pickle unless dill features are strictly needed"],"tags":["python","dependency","pickle","dill","installation"],"backgroundTag":"missing-optional-dependency","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}