{"record":{"id":"03d0e7b67b2b893d","repo":"hiyouga/LlamaFactory","slug":"deepspeed-zero-3-model-loading-bootstrap-failed-t","errorCode":null,"errorMessage":"DeepSpeed ZeRO-3 model-loading bootstrap failed: transformers still reports zero3 disabled after constructing HfDeepSpeedConfig. This usually means the runtime is using a different transformers installation than expected, or the DeepSpeed global state was not established correctly.","messagePattern":"DeepSpeed ZeRO-3 model-loading bootstrap failed: transformers still reports zero3 disabled after constructing HfDeepSpeedConfig\\. This usually means the runtime is using a different transformers installation than expected, or the DeepSpeed global state was not established correctly\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"src/llamafactory/v1/plugins/model_plugins/deepspeed_utils.py","lineNumber":137,"sourceCode":"    zero_stage = ds_config.get(\"zero_optimization\", {}).get(\"stage\")\n    if zero_stage != 3:\n        return None\n\n    # ZeRO-3 model loading needs concrete fp16/bf16 flags, not \"auto\".\n    mixed_precision = infer_deepspeed_mixed_precision(ds_config)\n\n    plugin = DeepSpeedPlugin(hf_ds_config=ds_config, zero3_init_flag=True)\n\n    if not plugin.hf_ds_config.is_zero3():\n        return None\n\n    # Reuse the same precision inference rule as the training-time DeepSpeed path\n    # so both model-loading and engine setup stay aligned.\n    plugin.set_mixed_precision(mixed_precision)\n    plugin.set_deepspeed_weakref()\n\n    if not _hf_is_deepspeed_zero3_enabled():\n        raise RuntimeError(\n            \"DeepSpeed ZeRO-3 model-loading bootstrap failed: transformers still reports zero3 disabled \"\n            \"after constructing HfDeepSpeedConfig. This usually means the runtime is using a different transformers \"\n            \"installation than expected, or the DeepSpeed global state was not established correctly.\"\n        )\n    return plugin\n\n\ndef teardown_deepspeed_zero3_model_loading(plugin) -> None:\n    if plugin is not None:\n        _unset_hf_deepspeed_config()\n","sourceCodeStart":119,"sourceCodeEnd":148,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/plugins/model_plugins/deepspeed_utils.py#L119-L148","documentation":"After building the accelerate DeepSpeedPlugin with zero3_init_flag=True and setting the weakref, the code asserts transformers' global is_deepspeed_zero3_enabled() is now True. If transformers still reports ZeRO-3 disabled, the bootstrap failed — typically a mismatched/duplicate transformers install or DeepSpeed global state never established (e.g. a second import path or an already-set HfDeepSpeedConfig).","triggerScenarios":"Multiple transformers versions on sys.path (editable install vs site-packages), a conda/pip mix where accelerate binds one and llamafactory another; HfDeepSpeedConfig global already torn down or never set because of import order; running inside a process that reset DeepSpeed state.","commonSituations":"Docker images with layered pip installs; `pip install -e .` over a released transformers; notebooks that re-initialize distributed state; mismatched accelerate/transformers/deepspeed versions after a partial upgrade.","solutions":["Verify a single transformers: `python -c \"import transformers; print(transformers.__file__, transformers.__version__)\"` and remove duplicates.","Reinstall the stack consistently in one environment: `pip install -U transformers accelerate deepspeed` (or `uv sync`).","Restart the process — this bootstrap must run once, before model creation; avoid re-running it after teardown in the same process.","Check deepspeed is importable and versions are compatible (deepspeed>=0.9.x with recent transformers).","If it persists, disable ZeRO-3 init (zero3_init_flag) so model loading uses the normal path."],"exampleFix":"# before (mixed envs)\npip install transformers==4.46.0  # while an editable copy shadows it\n\n# after\ngit clone ... && pip install -e .  # one canonical env\npython -c \"import transformers, accelerate, deepspeed; print(transformers.__file__, accelerate.__file__)\"  # same env prefix","handlingStrategy":"fallback","validationCode":"import transformers, accelerate, deepspeed, os\nroots = {os.path.dirname(m.__file__.rsplit('/', 1)[0]) for m in (transformers, accelerate, deepspeed)}\nprint(transformers.__version__, transformers.__file__)\nassert len(roots) <= 2, f'mixed site-packages: {roots}'  # tolerate one shared prefix","typeGuard":"def transformers_env_clean() -> bool:\n    \"\"\"True when a single transformers install is importable and deepspeed loads.\"\"\"\n    import importlib.util\n    specs = importlib.util.find_spec('transformers')\n    return specs is not None and deepspeed_is_importable()","tryCatchPattern":"try:\n    plugin = setup_deepspeed_zero3_model_loading()\nexcept RuntimeError as e:\n    if 'ZeRO-3 model-loading bootstrap failed' in str(e):\n        logger.warning('falling back to non-zero3-init loading; fix env')\n        plugin = None\n    else:\n        raise","preventionTips":["Pin and verify one transformers install per environment (no editable + released mix).","Print module __file__ in job logs to catch env drift.","Bootstrap ZeRO-3 init exactly once per process, before model construction.","Use fresh processes (not notebooks) for distributed DeepSpeed runs."],"tags":["deepspeed","environment","distributed","zero3","version-conflict"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}