Lightning-AI/pytorch-lightning · error · RuntimeError

The optimizer has references to the model's meta-device para

Error message

The optimizer has references to the model's meta-device parameters. Materializing them is is currently not supported unless you to set up the model and optimizer(s) separately. Create and set up the model first through `model = fabric.setup_module(model)`. Then create the optimizer and set it up: `optimizer = fabric.setup_optimizers(optimizer)`.

What it means

With FSDPStrategy, if an optimizer references model parameters that still live on the meta device (init_module/to_empty flow), Fabric cannot materialize them during a joint setup. The error directs you to set up the model first (materializing parameters) and create/set up the optimizer afterwards.

Source

Thrown at src/lightning/fabric/fabric.py:1211

    def _validate_launched(self) -> None:
        if not self._launched and not isinstance(self._strategy, (SingleDeviceStrategy, DataParallelStrategy)):
            raise RuntimeError(
                "To use Fabric with more than one device, you must call `.launch()` or use the CLI:"
                " `fabric run --help`."
            )

    def _validate_setup(self, module: nn.Module, optimizers: Sequence[Optimizer]) -> None:
        self._validate_launched()
        if isinstance(module, _FabricModule):
            raise ValueError("A model should be passed only once to the `setup` method.")

        if any(isinstance(opt, _FabricOptimizer) for opt in optimizers):
            raise ValueError("An optimizer should be passed only once to the `setup` method.")

        if isinstance(self._strategy, FSDPStrategy) and any(
            _has_meta_device_parameters_or_buffers(optimizer) for optimizer in optimizers
        ):
            raise RuntimeError(
                "The optimizer has references to the model's meta-device parameters. Materializing them is"
                " is currently not supported unless you to set up the model and optimizer(s) separately."
                " Create and set up the model first through `model = fabric.setup_module(model)`. Then create the"
                " optimizer and set it up: `optimizer = fabric.setup_optimizers(optimizer)`."
            )

    def _validate_setup_module(self, module: nn.Module) -> None:
        self._validate_launched()
        if isinstance(module, _FabricModule):
            raise ValueError("A model should be passed only once to the `setup_module` method.")

    def _validate_setup_optimizers(self, optimizers: Sequence[Optimizer]) -> None:
        self._validate_launched()
        if isinstance(self._strategy, (DeepSpeedStrategy, XLAStrategy)):
            raise RuntimeError(
                f"The `{type(self._strategy).__name__}` requires the model and optimizer(s) to be set up jointly"
                " through `.setup(model, optimizer, ...)`."
            )

View on GitHub (pinned to 9fed5c27d2)

Solutions

Reorder: model = fabric.setup_module(model) first, then optimizer = torch.optim.Adam(model.parameters(), lr=...); optimizer = fabric.setup_optimizers(optimizer)
Ensure parameters are moved to a real device (to_empty/device) before building the optimizer

Example fix

# before
with fabric.init_module():
    model = MyModel()
optimizer = Adam(model.parameters())
model, optimizer = fabric.setup(model, optimizer)  # FSDP
# after
with fabric.init_module():
    model = MyModel()
model = fabric.setup_module(model)
optimizer = fabric.setup_optimizers(Adam(model.parameters()))

Defensive patterns

Strategy: validation

Validate before calling

from lightning.fabric.strategies import FSDPStrategy
if isinstance(fabric.strategy, FSDPStrategy) and any(
    p.is_meta for opt in optimizers for group in opt.param_groups for p in group['params']
):
    raise SystemExit('set up the model first, then create the optimizer')

Prevention

With init_module + FSDP: call setup_module(model) BEFORE constructing the optimizer over model.parameters()

When it happens

Trigger: model = fabric.init_module(...) then creating an optimizer over model.parameters() (still meta) and calling fabric.setup(model, optimizer) with strategy='fsdp'.

Common situations: Using init_module for memory-efficient initialization (large LLMs) but constructing the optimizer before materialization; copying vanilla-Fabric init order into an FSDP script.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/564d3f2e6011a2ec. Report an issue: GitHub.