apache/beam · error · ValueError

Unknown pickler library

Error message

Unknown pickler library: {selected_library}

What it means

apache_beam.internal.pickler.set_library validates the requested pickle backend name. After resolving 'default' and checking dill/cloudpickle availability, any unrecognized library name falls through to a ValueError listing the invalid name. The accepted values are default, dill, dill_unsafe, and cloudpickle.

Solutions

1. Set --pickle_library to one of: default, dill, dill_unsafe, cloudpickle
2. Remove the pickle_library option entirely to use the default backend
3. If you meant stdlib pickle, use pickle_library=default
4. Check apache_beam.internal.pickler.USE_* constants for the exact accepted strings in your Beam version

Example fix

// before
--pickle_library=pickle   # ValueError: Unknown pickler library: pickle
// after
--pickle_library=default

ALLOWED = {'default', 'dill', 'dill_unsafe', 'cloudpickle'}
assert pickle_library in ALLOWED, f'invalid pickle_library: {pickle_library}'

try:
    pickler.set_library(lib)
except ValueError as e:
    logging.error('%s; falling back to default', e)
    pickler.set_library('default')

When it happens

Trigger: Passing pickle_library=<anything other than default|dill|dill_unsafe|cloudpickle> as a pipeline option, e.g. a typo like pickle_library=pickel, cPickle, or 'pickle'.

Common situations: Typoed pipeline option values; copying config from old Beam examples that used a renamed option value; assuming stdlib 'pickle' is a valid explicit choice (it is only reachable via 'default').

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• Basepath %r must be an Azure Blob Storage path.
• Cannot pickle closed files
• Cannot pickle file as it cannot be read
• Cannot pickle files that are not opened for reading
• Cannot pickle files that do not map to an actual file

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/29bbdc871851d789. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/internal/pickler.py:131

        "e.g. apache-beam[dill].")
  if selected_library == USE_DILL_UNSAFE and not dill_pickler:
    raise ImportError(
        "Pipeline option pickle_library=dill_unsafe is set, but dill is not "
        "installed. Install dill in job submission and runtime environments.")

  dill_is_requested = (
      selected_library == USE_DILL or selected_library == USE_DILL_UNSAFE)

  # If switching to or from dill, update the pickler hook overrides.
  if is_currently_dill() != dill_is_requested:
    dill_pickler.override_pickler_hooks(selected_library == USE_DILL)

  if dill_is_requested:
    desired_pickle_lib = dill_pickler
  elif selected_library == USE_CLOUDPICKLE:
    desired_pickle_lib = cloudpickle_pickler
  else:
    raise ValueError(f'Unknown pickler library: {selected_library}')

View on GitHub (pinned to 12126d8942)