{"record":{"id":"baca44f0dde54ec0","repo":"Lightning-AI/pytorch-lightning","slug":"empty-init-is-not-a-valid-choice-with-deepsp-baca44","errorCode":null,"errorMessage":"`{empty_init=}` is not a valid choice with `DeepSpeedStrategy` when ZeRO stage 3 is enabled.","messagePattern":"`(.+?)` is not a valid choice with `DeepSpeedStrategy` when ZeRO stage 3 is enabled\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/lightning/pytorch/strategies/deepspeed.py","lineNumber":533,"sourceCode":"        self.optimizers = [deepspeed_optimizer]\n\n        deepspeed_scheduler = model.lr_scheduler\n        if deepspeed_scheduler is not None:\n            # disable deepspeed lr scheduling as lightning manages scheduling\n            model.lr_scheduler = None\n            if lr_scheduler is None:\n                lr_scheduler = LRSchedulerConfig(deepspeed_scheduler, interval=\"step\")\n            else:\n                lr_scheduler.scheduler = deepspeed_scheduler\n            self.lr_scheduler_configs = [lr_scheduler]\n        self.model = model\n\n    @contextmanager\n    @override\n    def tensor_init_context(self, empty_init: Optional[bool] = None) -> Generator[None, None, None]:\n        if self.zero_stage_3:\n            if empty_init is False:\n                raise NotImplementedError(\n                    f\"`{empty_init=}` is not a valid choice with `DeepSpeedStrategy` when ZeRO stage 3 is enabled.\"\n                )\n            yield\n            return\n        with super().tensor_init_context(empty_init=empty_init):\n            yield\n\n    @contextmanager\n    @override\n    def model_sharded_context(self) -> Generator[None, None, None]:\n        import deepspeed\n\n        self._init_config_if_needed()\n        with deepspeed.zero.Init(\n            enabled=self.zero_stage_3,\n            remote_device=self.remote_device,\n            config_dict_or_path=self.config,\n        ):","sourceCodeStart":515,"sourceCodeEnd":551,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/strategies/deepspeed.py#L515-L551","documentation":"Under ZeRO stage 3, model parameters are partitioned across ranks at creation time, so tensors cannot be materialized with real values on every rank during `__init__`/`configure_model`. `empty_init=False` (which requests non-empty, real initialization) therefore contradicts ZeRO-3 and raises NotImplementedError in tensor_init_context.","triggerScenarios":"`DeepSpeedStrategy(config with zero_optimization stage=3)` combined with Lightning's `TensorInitContext` requesting `empty_init=False` — typically set via `Trainer(..., plugins=...)` or Fabric's `init_tensor_workers`/model init with empty_init=False.","commonSituations":"Users passing `empty_init=False` to get deterministic real init for debugging while enabling ZeRO-3; migrating from FSDP-style configs where empty_init is tunable.","solutions":["Use the default `empty_init=None`/`True` with ZeRO stage 3 and initialize weights in `configure_model()` or after setup","If real initialization is required, lower the ZeRO stage (e.g. stage 2)","Seed-based determinism: keep empty init and set PL_GLOBAL_SEED so post-init re-initialization is reproducible"],"exampleFix":"# before\nstrategy = DeepSpeedStrategy(config={\"zero_optimization\": {\"stage\": 3}})\nmodel = MyModel(..., empty_init=False)\n\n# after\nstrategy = DeepSpeedStrategy(config={\"zero_optimization\": {\"stage\": 3}})\nmodel = MyModel(...)  # init under empty context; real init in configure_model()","handlingStrategy":"validation","validationCode":"cfg = {\"zero_optimization\": {\"stage\": 3}}\nempty_init = None  # never pass False under ZeRO-3\nmodel = L.LightningModule(...)  # rely on default empty init","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Do weight initialization inside configure_model() or on_configure_sharded_model so it runs after partitioning","Don't copy empty_init=False flags from non-ZeRO-3 codepaths"],"tags":["deepspeed","zero-stage-3","weight-initialization"],"backgroundTag":"zero-stage-3-init-conflict","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}