{"record":{"id":"bb86f9323208c024","repo":"apache/beam","slug":"recursive-id-generation-detected-for-class-def-the-id","errorCode":null,"errorMessage":"Recursive ID generation detected for {class_def}. The id_generator cannot recursively request an ID for the same class.","messagePattern":"Recursive ID generation detected for (.+?)\\. The id_generator cannot recursively request an ID for the same class\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py","lineNumber":189,"sourceCode":"  get_code_object_params: typing.Optional[GetCodeObjectParams] = None\n  pickle_main_by_ref: bool = False\n\n\nDEFAULT_CONFIG = CloudPickleConfig()\n_GENERATING_SENTINEL = object()\nbuiltin_code_type = None\nif PYPY:\n  # builtin-code objects only exist in pypy\n  builtin_code_type = type(float.__new__.__code__)\n\n_extract_code_globals_cache = weakref.WeakKeyDictionary()\n\n\ndef _get_or_create_tracker_id(class_def, id_generator):\n  with _DYNAMIC_CLASS_TRACKER_LOCK:\n    class_tracker_id = _DYNAMIC_CLASS_TRACKER_BY_CLASS.get(class_def)\n    if class_tracker_id is _GENERATING_SENTINEL and id_generator:\n      raise RuntimeError(\n          f\"Recursive ID generation detected for {class_def}. \"\n          f\"The id_generator cannot recursively request an ID for the same class.\"\n      )\n\n    if class_tracker_id is None and id_generator is not None:\n      _DYNAMIC_CLASS_TRACKER_BY_CLASS[class_def] = _GENERATING_SENTINEL\n      try:\n        class_tracker_id = id_generator(class_def)\n        _DYNAMIC_CLASS_TRACKER_BY_CLASS[class_def] = class_tracker_id\n        _DYNAMIC_CLASS_TRACKER_BY_ID[class_tracker_id] = class_def\n      except Exception:\n        _DYNAMIC_CLASS_TRACKER_BY_CLASS.pop(class_def, None)\n        raise\n  return class_tracker_id\n\n\ndef _lookup_class_or_track(class_tracker_id, class_def):\n  if class_tracker_id is not None:","sourceCodeStart":171,"sourceCodeEnd":207,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py#L171-L207","documentation":"Beam's cloudpickle tracks dynamically created classes with ID generation; _get_or_create_tracker_id sets a sentinel while generating an ID. If, during that generation, the same class is requested again (recursive class definition, e.g. a class whose getnewargs or TypeVar decomposition references itself), the recursion is detected and RuntimeError is raised to prevent infinite recursion.","triggerScenarios":"Pickling dynamically generated classes (namedtuples, Enums, generic TypeVar-parameterized classes) where _decompose_typevar/_class_getnewargs/_enum_getnewargs re-enter ID generation for the same class_def; self-referential class definitions created at runtime.","commonSituations":"Serializing pipelines containing recursively defined dynamic classes or self-referential generics with Beam's Python SDK; pickling a class created inside a DoFn that refers to itself in its own __reduce__/__getnewargs__.","solutions":["Restructure the dynamic class so it does not reference itself during construction/getnewargs (break the recursion)","Define the class statically at module level instead of dynamically so it is not tracked by the ID generator","Avoid pickling the recursive object directly; pickle a serializable surrogate (e.g. a spec dict) and rebuild on load","If caused by a Beam/cloudpickle bug, report with a reproducer and pin a Beam version where the tracker behaves correctly"],"exampleFix":"// before\nclass Node(namedtuple('Node', 'children')):\n  def __getnewargs__(self): return (Node([]),)  # self-reference during ID gen\n// after\nclass Node(namedtuple('Node', 'children')):\n  def __getnewargs__(self): return (list(self.children),)","handlingStrategy":"try-catch","validationCode":"# detect self-referential dynamic classes before pickling\ndef references_itself(cls):\n    import inspect\n    return any(cls in getattr(a, '__globals__', {}).values() or a is cls\n               for a in getattr(cls, '__reduce__', lambda: ())()) or cls in repr(getattr(cls, '__getnewargs__', lambda: ())())","typeGuard":"def is_picklable_dynamically(cls) -> bool:\n    return not (hasattr(cls, '__getnewargs__') and cls in cls.__getnewargs__())","tryCatchPattern":"import cloudpickle\ntry:\n    data = cloudpickle.dumps(obj)\nexcept RuntimeError as e:\n    if 'Recursive ID generation' in str(e):\n        data = cloudpickle.dumps(obj.__reduce__()[1])  # serialize surrogate args instead\n    else:\n        raise","preventionTips":["Avoid self-referential dynamically generated classes (namedtuples/Enums) in DoFns","Define classes statically at module level so they bypass the dynamic-class tracker","Break recursive class references with surrogates or lazy rebuild hooks","Pin and test Beam versions when relying on dynamic class serialization"],"tags":["python","apache-beam","cloudpickle","serialization","recursion"],"backgroundTag":"internal-invariant-violation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}