Lightning-AI/pytorch-lightning · error · NotImplementedError

Calling `trainer.fit()` twice on the same Trainer instance u

Error message

Calling `trainer.fit()` twice on the same Trainer instance using a spawn-based strategy is not supported. You can work around this limitation by creating a new Trainer instance and passing the `fit(ckpt_path=...)` argument.

What it means

Same restriction as the multiprocessing spawn launcher: the XLA launcher (fork/spawn based) cannot relaunch a second fit() on the same Trainer instance, because trainer state cannot be safely restored across repeated XLA multiprocess launches (issue #18775).

Source

Thrown at src/lightning/pytorch/strategies/launchers/xla.py:80

    @override
    def launch(self, function: Callable, *args: Any, trainer: Optional["pl.Trainer"] = None, **kwargs: Any) -> Any:
        """Launches processes that run the given function in parallel.

        The function is allowed to have a return value. However, when all processes join, only the return value
        of worker process 0 gets returned from this `launch` method in the main process.

        Arguments:
            function: The entry point for all launched processes.
            *args: Optional positional arguments to be passed to the given function.
            trainer: Optional reference to the :class:`~lightning.pytorch.trainer.trainer.Trainer` for which
                a selected set of attributes get restored in the main process after processes join.
            **kwargs: Optional keyword arguments to be passed to the given function.

        """
        if self._already_fit and trainer is not None and trainer.state.fn == TrainerFn.FITTING:
            # resolving https://github.com/Lightning-AI/pytorch-lightning/issues/18775 will lift this restriction
            raise NotImplementedError(
                "Calling `trainer.fit()` twice on the same Trainer instance using a spawn-based strategy is not"
                " supported. You can work around this by creating a new Trainer instance and passing the"
                " `fit(ckpt_path=...)` argument."
            )

        # pjrt requires that the queue is serializable
        return_queue = mp.Manager().Queue()

        import torch_xla.distributed.xla_multiprocessing as xmp

        spawn_kwargs = {}
        nprocs = self._strategy.num_processes
        if nprocs == 1:
            # avoid warning: "Unsupported nprocs". If it's 1, it will call the launched function directly.
            # otherwise it will use all devices
            spawn_kwargs["nprocs"] = nprocs

        process_context = xmp.spawn(

View on GitHub (pinned to 9fed5c27d2)

Solutions

Create a new Trainer instance and pass ckpt_path to resume
Combine both stages into one fit() call
Track Lightning-AI/pytorch-lightning#18775

Example fix

# before
trainer.fit(model)
trainer.fit(model, ckpt_path="last.ckpt")  # NotImplementedError

# after
trainer.fit(model)
new_trainer = L.Trainer(max_epochs=10)
new_trainer.fit(model, ckpt_path="last.ckpt")

Defensive patterns

Strategy: fallback

Validate before calling

if getattr(trainer.strategy.launcher, "_already_fit", False):
    trainer = L.Trainer(max_epochs=..., plugins=...)  # new trainer, resume via ckpt_path

Prevention

One Trainer per fit() for XLA/TPU runs
Resume with fit(ckpt_path=...) on the fresh Trainer

When it happens

Trigger: Calling trainer.fit() a second time with trainer.state.fn == FITTING while the strategy uses the XLA launcher and _already_fit is True (e.g. XLAStrategy on TPU).

Common situations: Fine-tuning or continuing training on TPU with the same Trainer; two-stage training scripts calling fit twice.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/d64a1df0c0294de1. Report an issue: GitHub.