Lightning-AI/pytorch-lightning · error · RuntimeError

The optimizer has references to the model's meta-device para

Error message

The optimizer has references to the model's meta-device parameters. Materializing them is is currently not supported. Create the optimizer after setting up the model, then call `fabric.setup_optimizers(optimizer)`.

What it means

Raised when an optimizer passed to `fabric.setup_optimizers()` references model parameters that still live on the meta device (e.g. created via `torch.device('meta')` or a factory like `with torch.device('meta'):`). Fabric cannot materialize meta parameters through this path, so the optimizer must be created after the model is set up (which materializes the parameters).

Source

Thrown at src/lightning/fabric/fabric.py:1238

        if isinstance(module, _FabricModule):
            raise ValueError("A model should be passed only once to the `setup_module` method.")

    def _validate_setup_optimizers(self, optimizers: Sequence[Optimizer]) -> None:
        self._validate_launched()
        if isinstance(self._strategy, (DeepSpeedStrategy, XLAStrategy)):
            raise RuntimeError(
                f"The `{type(self._strategy).__name__}` requires the model and optimizer(s) to be set up jointly"
                " through `.setup(model, optimizer, ...)`."
            )

        if not optimizers:
            raise ValueError("`setup_optimizers` requires at least one optimizer as input.")

        if any(isinstance(opt, _FabricOptimizer) for opt in optimizers):
            raise ValueError("An optimizer should be passed only once to the `setup_optimizers` method.")

        if any(_has_meta_device_parameters_or_buffers(optimizer) for optimizer in optimizers):
            raise RuntimeError(
                "The optimizer has references to the model's meta-device parameters. Materializing them is"
                " is currently not supported. Create the optimizer after setting up the model, then call"
                " `fabric.setup_optimizers(optimizer)`."
            )

    def _validate_setup_dataloaders(self, dataloaders: Sequence[DataLoader]) -> None:
        self._validate_launched()
        if not dataloaders:
            raise ValueError("`setup_dataloaders` requires at least one dataloader as input.")

        if any(isinstance(dl, _FabricDataLoader) for dl in dataloaders):
            raise ValueError("A dataloader should be passed only once to the `setup_dataloaders` method.")

        if any(not isinstance(dl, DataLoader) for dl in dataloaders):
            raise TypeError("Only PyTorch DataLoader are currently supported in `setup_dataloaders`.")

    @staticmethod
    def _configure_callbacks(callbacks: Optional[Union[list[Any], Any]]) -> list[Any]:

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Create the optimizer AFTER `model = fabric.setup(model)` so parameters are materialized, then call `fabric.setup_optimizers(optimizer)`
2. Ensure the strategy you use supports meta-device init at all; for strategies requiring joint setup, use `fabric.setup(model, optimizer)` instead of the split path

Example fix

# before
with torch.device('meta'):
    model = BigModel()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)  # meta params
model = fabric.setup(model)
fabric.setup_optimizers(optimizer)

# after
with torch.device('meta'):
    model = BigModel()
model = fabric.setup(model)  # materializes params
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
fabric.setup_optimizers(optimizer)

def has_meta_params(module) -> bool:
    return any(p.is_meta for p in module.parameters())
assert not has_meta_params(model), 'materialize model (fabric.setup) before creating optimizer'

When it happens

Trigger: Creating a model under `with torch.device('meta'):`, building an optimizer over `model.parameters()` while still on meta device, then calling `fabric.setup(model)` followed by `fabric.setup_optimizers(optimizer)`. The check `_has_meta_device_parameters_or_buffers` finds meta-device params in the optimizer state.

Common situations: Using meta-device initialization to avoid allocating memory twice for large models (FSDP/deferred init workflows) and keeping the old combined setup call order; upgrading Fabric versions where split setup became required for this pattern.

Related errors

• The optimizer has references to the model's meta-device para
• The hybrid sharding strategy requires you to pass at least o
• `setup_optimizers` requires at least one optimizer as input.
• An optimizer should be passed only once to the `setup_optimi
• `setup_dataloaders` requires at least one dataloader as inpu

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/dc85737e1c6fc8ca. Report an issue: GitHub.