Lightning-AI/pytorch-lightning · error · NotImplementedError

Calling `trainer.fit()` twice on the same Trainer instance u

Error message

Calling `trainer.fit()` twice on the same Trainer instance using a spawn-based strategy is not supported. You can work around this limitation by creating a new Trainer instance and passing the `fit(ckpt_path=...)` argument.

What it means

A spawn-based multiprocessing launcher cannot be reused for a second fit run on the same Trainer: after the first fit, the spawned processes and the pickled trainer state make a relaunch unsafe, so Lightning blocks it with NotImplementedError (tracked in issue #18775).

Source

Thrown at src/lightning/pytorch/strategies/launchers/multiprocessing.py:115

        The function is allowed to have a return value. However, when all processes join, only the return value
        of worker process 0 gets returned from this `launch` method in the main process.

        Arguments:
            function: The entry point for all launched processes.
            *args: Optional positional arguments to be passed to the given function.
            trainer: Optional reference to the :class:`~lightning.pytorch.trainer.trainer.Trainer` for which
                a selected set of attributes get restored in the main process after processes join.
            **kwargs: Optional keyword arguments to be passed to the given function.

        """
        if self._start_method in ("fork", "forkserver"):
            _check_bad_cuda_fork()
        if self._start_method == "spawn":
            _check_missing_main_guard()
        if self._already_fit and trainer is not None and trainer.state.fn == TrainerFn.FITTING:
            # resolving https://github.com/Lightning-AI/pytorch-lightning/issues/18775 will lift this restriction
            raise NotImplementedError(
                "Calling `trainer.fit()` twice on the same Trainer instance using a spawn-based strategy is not"
                " supported. You can work around this limitation by creating a new Trainer instance and passing the"
                " `fit(ckpt_path=...)` argument."
            )

        # The default cluster environment in Lightning chooses a random free port number
        # This needs to be done in the main process here before starting processes to ensure each rank will connect
        # through the same port
        assert self._strategy.cluster_environment is not None
        os.environ["MASTER_PORT"] = str(self._strategy.cluster_environment.main_port)

        context = mp.get_context(self._start_method)
        return_queue = context.SimpleQueue()

        if self._start_method == "spawn":
            global_states = _GlobalStateSnapshot.capture()
            process_args = [trainer, function, args, kwargs, return_queue, global_states]
        else:

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Create a fresh Trainer for the second fit and pass fit(ckpt_path=...) to resume
2. Restructure to a single fit() call (use callbacks/loops to change behavior mid-training)
3. Track the issue Lightning-AI/pytorch-lightning#18775 for a version where the restriction is lifted

Example fix

# before
trainer.fit(model)
trainer.fit(model, ckpt_path="last.ckpt")  # NotImplementedError

# after
trainer.fit(model)
trainer2 = L.Trainer(max_epochs=20)
trainer2.fit(model, ckpt_path="last.ckpt")

if getattr(trainer.strategy.launcher, "_already_fit", False):
    trainer = L.Trainer(**trainer_flags)  # fresh instance, pass ckpt_path to fit

When it happens

Trigger: Calling trainer.fit() a second time (e.g. a second epoch sweep or a fine-tune stage) on the same Trainer whose strategy uses the spawn-based _MultiProcessingLauncher, with trainer.state.fn == FITTING.

Common situations: Fine-tuning loops, hyperparameter sweeps reusing one Trainer, resumed training scripts that call fit twice (train then continue training).

Related errors

• Device should be CUDA, got {device} instead.
• You requested to find {num_devices} devices but there are no
• You requested to find {num_devices} devices but this machine
• You requested to find {num_devices} devices but only {len(av
• Device should be MPS, got {device} instead.

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/b58fe664b6148a38. Report an issue: GitHub.