apache/beam · error · RuntimeError

Recursive ID generation detected for {class_def}. The id_gen

Error message

Recursive ID generation detected for {class_def}. The id_generator cannot recursively request an ID for the same class.

What it means

Beam's cloudpickle tracks dynamically created classes with ID generation; _get_or_create_tracker_id sets a sentinel while generating an ID. If, during that generation, the same class is requested again (recursive class definition, e.g. a class whose getnewargs or TypeVar decomposition references itself), the recursion is detected and RuntimeError is raised to prevent infinite recursion.

Source

Thrown at sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:189

  get_code_object_params: typing.Optional[GetCodeObjectParams] = None
  pickle_main_by_ref: bool = False


DEFAULT_CONFIG = CloudPickleConfig()
_GENERATING_SENTINEL = object()
builtin_code_type = None
if PYPY:
  # builtin-code objects only exist in pypy
  builtin_code_type = type(float.__new__.__code__)

_extract_code_globals_cache = weakref.WeakKeyDictionary()


def _get_or_create_tracker_id(class_def, id_generator):
  with _DYNAMIC_CLASS_TRACKER_LOCK:
    class_tracker_id = _DYNAMIC_CLASS_TRACKER_BY_CLASS.get(class_def)
    if class_tracker_id is _GENERATING_SENTINEL and id_generator:
      raise RuntimeError(
          f"Recursive ID generation detected for {class_def}. "
          f"The id_generator cannot recursively request an ID for the same class."
      )

    if class_tracker_id is None and id_generator is not None:
      _DYNAMIC_CLASS_TRACKER_BY_CLASS[class_def] = _GENERATING_SENTINEL
      try:
        class_tracker_id = id_generator(class_def)
        _DYNAMIC_CLASS_TRACKER_BY_CLASS[class_def] = class_tracker_id
        _DYNAMIC_CLASS_TRACKER_BY_ID[class_tracker_id] = class_def
      except Exception:
        _DYNAMIC_CLASS_TRACKER_BY_CLASS.pop(class_def, None)
        raise
  return class_tracker_id


def _lookup_class_or_track(class_tracker_id, class_def):
  if class_tracker_id is not None:

View on GitHub (pinned to 12126d8942)

Solutions

Restructure the dynamic class so it does not reference itself during construction/getnewargs (break the recursion)
Define the class statically at module level instead of dynamically so it is not tracked by the ID generator
Avoid pickling the recursive object directly; pickle a serializable surrogate (e.g. a spec dict) and rebuild on load
If caused by a Beam/cloudpickle bug, report with a reproducer and pin a Beam version where the tracker behaves correctly

Example fix

// before
class Node(namedtuple('Node', 'children')):
  def __getnewargs__(self): return (Node([]),)  # self-reference during ID gen
// after
class Node(namedtuple('Node', 'children')):
  def __getnewargs__(self): return (list(self.children),)

Defensive patterns

Strategy: try-catch

Validate before calling

# detect self-referential dynamic classes before pickling
def references_itself(cls):
    import inspect
    return any(cls in getattr(a, '__globals__', {}).values() or a is cls
               for a in getattr(cls, '__reduce__', lambda: ())()) or cls in repr(getattr(cls, '__getnewargs__', lambda: ())())

Type guard

def is_picklable_dynamically(cls) -> bool:
    return not (hasattr(cls, '__getnewargs__') and cls in cls.__getnewargs__())

Try / catch

import cloudpickle
try:
    data = cloudpickle.dumps(obj)
except RuntimeError as e:
    if 'Recursive ID generation' in str(e):
        data = cloudpickle.dumps(obj.__reduce__()[1])  # serialize surrogate args instead
    else:
        raise

Prevention

Avoid self-referential dynamically generated classes (namedtuples/Enums) in DoFns
Define classes statically at module level so they bypass the dynamic-class tracker
Break recursive class references with surrogates or lazy rebuild hooks
Pin and test Beam versions when relying on dynamic class serialization

When it happens

Trigger: Pickling dynamically generated classes (namedtuples, Enums, generic TypeVar-parameterized classes) where _decompose_typevar/_class_getnewargs/_enum_getnewargs re-enter ID generation for the same class_def; self-referential class definitions created at runtime.

Common situations: Serializing pipelines containing recursively defined dynamic classes or self-referential generics with Beam's Python SDK; pickling a class created inside a DoFn that refers to itself in its own __reduce__/__getnewargs__.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/bb86f9323208c024. Report an issue: GitHub.