{"record":{"id":"50be67d0ee643d6c","repo":"xai-org/x-algorithm","slug":"orbax-version-is-too-old","errorCode":null,"errorMessage":"Orbax version is too old","messagePattern":"Orbax version is too old","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"phoenix/xrex/utils/checkpointing.py","lineNumber":97,"sourceCode":"    if _CHECKPOINTER is None:\n        handler_kwargs: dict[str, Any] = {\"use_zarr3\": True, \"restore_concurrent_gb\": 1}\n        if save_concurrent_gb is not None:\n            handler_kwargs[\"save_concurrent_gb\"] = save_concurrent_gb\n            handler_kwargs[\"restore_concurrent_gb\"] = save_concurrent_gb\n        _CHECKPOINTER_SAVE_CONCURRENT_GB = save_concurrent_gb\n        rank_logger.info(\n            \"Creating AsyncCheckpointer: PyTreeCheckpointHandler(use_zarr3=True, \"\n            \"save_concurrent_gb=%s, restore_concurrent_gb=%s) \"\n            \"[None => Orbax write-limiter default 96GB; D2H still all-at-once \"\n            \"unless save_checkpoint registers ThrottledD2HArrayHandler]\",\n            save_concurrent_gb,\n            save_concurrent_gb,\n        )\n        _CHECKPOINTER = ocp.AsyncCheckpointer(\n            ocp.PyTreeCheckpointHandler(**handler_kwargs), timeout_secs\n        )\n        if not hasattr(_CHECKPOINTER, \"_post_finalization_callback\"):\n            raise RuntimeError(\"Orbax version is too old\")\n    elif save_concurrent_gb is not None and _CHECKPOINTER_SAVE_CONCURRENT_GB != save_concurrent_gb:\n        rank_logger.warning(\n            \"get_checkpointer(save_concurrent_gb=%s) ignored; checkpointer already \"\n            \"created with save_concurrent_gb=%s. D2H throttle (if enabled) still \"\n            \"uses the value passed to save_checkpoint.\",\n            save_concurrent_gb,\n            _CHECKPOINTER_SAVE_CONCURRENT_GB,\n        )\n    return _CHECKPOINTER\n\n\nclass NoCompressionArrayHandler(ocp.type_handlers.ArrayHandler):\n    def _get_json_tspec_write(self, *args, **kwargs):\n        spec = super()._get_json_tspec_write(*args, **kwargs)\n        for codec in spec[\"metadata\"][\"codecs\"]:\n            cfg = codec[\"configuration\"]\n            cfg[\"codecs\"] = [c for c in cfg[\"codecs\"] if c[\"name\"] != \"zstd\"]\n        return spec","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/utils/checkpointing.py#L79-L115","documentation":"get_checkpointer builds an Orbax AsyncCheckpointer and requires the private _post_finalization_callback attribute to exist; older Orbax releases lack it, so the code refuses to run rather than silently skipping finalization callbacks. It's a hard version gate on the orbax-checkpoint dependency.","triggerScenarios":"Installing a version of orbax-checkpoint older than the one this repo was developed against, then calling save_checkpoint (which calls get_checkpointer).","commonSituations":"Fresh environment resolving to an old orbax from a transitive dependency; pip install orbax-checkpoint without a lower bound; environment drift after a base-image update.","solutions":["Upgrade orbax-checkpoint to the version pinned in the repo's requirements/pyproject (one that has _post_finalization_callback)","Verify with: python -c \"import orbax.checkpoint as ocp; print(ocp.__version__)\" and compare against the pin","Recreate the environment from the project's lockfile"],"exampleFix":"# before\npip install orbax-checkpoint  # resolves to old version\n# after\npip install 'orbax-checkpoint>=0.5.0'  # or the repo's pinned version","handlingStrategy":"validation","validationCode":"import orbax.checkpoint as ocp\nc = ocp.AsyncCheckpointer(ocp.PyTreeCheckpointHandler())\nassert hasattr(c, '_post_finalization_callback'), 'orbax too old; upgrade'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pin orbax-checkpoint in requirements/lockfile with a tested lower bound","Add a startup version check for orbax before training begins","Recreate environments from the lockfile rather than ad-hoc installs"],"tags":["orbax","version-incompatibility","checkpointing","dependency"],"backgroundTag":"dependency-version-too-old","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}