{"record":{"id":"d41ea34420f3e1cf","repo":"Lightning-AI/pytorch-lightning","slug":"empty-init-is-not-a-valid-choice-with-deepsp","errorCode":null,"errorMessage":"`{empty_init=}` is not a valid choice with `DeepSpeedStrategy` when ZeRO stage 3 is enabled.","messagePattern":"`(.+?)` is not a valid choice with `DeepSpeedStrategy` when ZeRO stage 3 is enabled\\.","errorType":"validation","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/strategies/deepspeed.py","lineNumber":381,"sourceCode":"        For training, see :meth:`setup_module_and_optimizers`.\n\n        \"\"\"\n        self._deepspeed_engine, _, _ = self._initialize_engine(module)\n        return self._deepspeed_engine\n\n    @override\n    def setup_optimizer(self, optimizer: Optimizer) -> Optimizer:\n        \"\"\"Optimizers can only be set up jointly with the model in this strategy.\n\n        Please use :meth:`setup_module_and_optimizers` to set up both module and optimizer together.\n\n        \"\"\"\n        raise NotImplementedError(self._err_msg_joint_setup_required())\n\n    @override\n    def module_init_context(self, empty_init: Optional[bool] = None) -> AbstractContextManager:\n        if self.zero_stage_3 and empty_init is False:\n            raise NotImplementedError(\n                f\"`{empty_init=}` is not a valid choice with `DeepSpeedStrategy` when ZeRO stage 3 is enabled.\"\n            )\n        module_sharded_ctx = self.module_sharded_context()\n        stack = ExitStack()\n        if not self.zero_stage_3:\n            stack.enter_context(super().module_init_context(empty_init=empty_init))\n        stack.enter_context(module_sharded_ctx)\n        return stack\n\n    @override\n    def module_sharded_context(self) -> AbstractContextManager:\n        # Current limitation in Fabric: The config needs to be fully determined at the time of calling the context\n        # manager. Later modifications through e.g. `Fabric.setup()` won't have an effect here.\n\n        import deepspeed\n\n        assert self._config_initialized\n        return deepspeed.zero.Init(","sourceCodeStart":363,"sourceCodeEnd":399,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/strategies/deepspeed.py#L363-L399","documentation":"With DeepSpeed ZeRO stage 3, model parameters are partitioned at creation time, so the module must be materialized inside DeepSpeed's init context — you cannot pre-create a fully-initialized module. module_init_context therefore rejects empty_init=False when zero_stage_3 is enabled.","triggerScenarios":"DeepSpeedStrategy(stage=3) combined with fabric.setup_module(module_created_with_empty_init=False), i.e. passing a module built outside the init context / with meta or normal init while ZeRO-3 partitioning is on; also explicitly passing empty_init=False to module_init_context.","commonSituations":"Loading a pretrained model then handing it to a ZeRO-3 strategy; code that instantiates the model before fabric.setup and requests real (non-empty) init.","solutions":["Create the model inside fabric's init context: with fabric.init_module(): model = MyModel() so ZeRO-3 can partition it at creation","Keep zero_stage < 3 if you must pass an already-materialized module","For pretrained weights under ZeRO-3, use the deepspeed checkpoint load path or load state after engine init"],"exampleFix":"# before\nmodel = MyModel()  # materialized outside\nstrategy = DeepSpeedStrategy(stage=3)\nfabric.setup(model, opt)\n# after\nstrategy = DeepSpeedStrategy(stage=3)\nfabric = Fabric(strategy=strategy)\nwith fabric.init_module(empty_init=True):\n    model = MyModel()\nmodel, opt = fabric.setup(model, opt)","handlingStrategy":"validation","validationCode":"from lightning.fabric.strategies.deepspeed import DeepSpeedStrategy\nif isinstance(fabric.strategy, DeepSpeedStrategy) and fabric.strategy.zero_stage_3:\n    with fabric.init_module(empty_init=True):\n        model = MyModel()  # create inside the context\nelse:\n    model = MyModel()","typeGuard":"from lightning.fabric.strategies.deepspeed import DeepSpeedStrategy\n\ndef must_init_in_context(strategy) -> bool:\n    return isinstance(strategy, DeepSpeedStrategy) and strategy.zero_stage_3","tryCatchPattern":"try:\n    ctx = fabric.strategy.module_init_context(empty_init=False)\nexcept NotImplementedError:\n    ctx = fabric.strategy.module_init_context(empty_init=True)","preventionTips":["Always create big models inside fabric.init_module()","Never pass pre-materialized modules to a ZeRO-3 strategy"],"tags":["deepspeed","zero-3","model-init","pytorch-lightning"],"backgroundTag":"unsupported-operation-combination","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}