Lightning-AI/pytorch-lightning · error · Exception

The server didn't start within {self.timeout} seconds.

Error message

The server didn't start within {self.timeout} seconds.

What it means

ServableModuleValidator spawns a subprocess running a FastAPI server on host:port and polls /ping until it returns 200. If the server is not ready within self.timeout seconds (default 30), the process is killed and a generic Exception is raised.

Source

Thrown at src/lightning/pytorch/serve/servable_module_validator.py:111

        # Note: The Trainer needs to be detached from the pl_module before starting the process.
        # This would fail during the deepcopy with DDP.
        servable_module.trainer = None

        process = Process(target=self._start_server, args=(servable_module, self.host, self.port, self.optimization))
        process.start()

        servable_module.trainer = trainer

        ready = False
        t0 = time.time()
        while not ready:
            with contextlib.suppress(requests.exceptions.ConnectionError):
                resp = requests.get(f"http://{self.host}:{self.port}/ping")
                ready = resp.status_code == 200
            if time.time() - t0 > self.timeout:
                process.kill()
                raise Exception(f"The server didn't start within {self.timeout} seconds.")
            time.sleep(0.1)

        payload = servable_module.configure_payload()

        if "body" not in payload:
            raise Exception(f'Your provided payload {payload} should have a field named "body".')

        self.resp = requests.post(f"http://{self.host}:{self.port}/serve", json=payload)
        process.kill()

        if is_overridden("configure_response", servable_module, ServableModule):
            response = servable_module.configure_response()
            if self.resp.json() != response:
                raise Exception(f"The expected response {response} doesn't match the generated one {self.resp.json()}.")

        if self.exit_on_failure and not self.successful:
            raise MisconfigurationException("The model isn't servable. Investigate the traceback and try again.")

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Increase the timeout: ServableModuleValidator(host=..., port=..., timeout=120)
2. Check the port is free (change port or kill the holder: lsof -i :PORT)
3. Watch subprocess logs for the real startup failure (the validator only reports the timeout)
4. Pre-warm heavy imports/weights so the subprocess starts faster

Example fix

// before
trainer = Trainer(callbacks=[ServableModuleValidator(host="127.0.0.1", port=8000)])
// after
trainer = Trainer(callbacks=[ServableModuleValidator(host="127.0.0.1", port=8001, timeout=120)])

import socket

with socket.socket() as s:
    s.bind(("127.0.0.1", 8000))  # raises if port taken — pick another port before training

try:
    trainer.fit(model)
except Exception as e:
    if "didn't start within" in str(e):
        validator = ServableModuleValidator(host=h, port=p, timeout=120)
        trainer = Trainer(callbacks=[validator], max_epochs=trainer.max_epochs)
        trainer.fit(model)  # retry with longer timeout / different port
    else:
        raise

When it happens

Trigger: Slow model load (large weights on slow disk), port already occupied so the server never binds, slow first import of torch/FastAPI in the subprocess, or an exception during server startup that prevents /ping from ever answering.

Common situations: CI machines with cold caches, another process holding the chosen port, firewalls blocking localhost connections, or an error inside the server subprocess whose traceback is only visible in subprocess logs.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• The `configure_payload` method needs to be overridden.
• The `configure_serialization` method needs to be overridden.
• The `serve_step` method needs to be overridden.
• Your provided payload {payload} should have a field named "b
• The expected response {response} doesn't match the generated

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/aaf184e4c1d2aa40. Report an issue: GitHub.