Unity-Technologies/ml-agents · error · TimeoutError

Workers {still_waiting} stuck in waiting state

Error message

Workers {still_waiting} stuck in waiting state

What it means

A sentinel error raised by SubprocessEnvManager._drain_step_queue (invoked from _restart_failed_workers) when, after waiting up to a one-minute deadline, some environment workers are still flagged as 'waiting' — i.e. they were sent a step/reset command but never produced a response on the step queue. It means those worker processes or their Unity environments hung or died without sending ENV_EXITED, so the manager cannot cleanly collect their state before restarting them.

Source

Thrown at ml-agents/mlagents/trainers/subprocess_env_manager.py:352

        """
        all_failures = {}
        workers_still_pending = {w.worker_id for w in self.env_workers if w.waiting}
        deadline = datetime.datetime.now() + datetime.timedelta(minutes=1)
        while workers_still_pending and deadline > datetime.datetime.now():
            try:
                while True:
                    step: EnvironmentResponse = self.step_queue.get_nowait()
                    if step.cmd == EnvironmentCommand.ENV_EXITED:
                        workers_still_pending.add(step.worker_id)
                        all_failures[step.worker_id] = step.payload
                    else:
                        workers_still_pending.remove(step.worker_id)
                        self.env_workers[step.worker_id].waiting = False
            except EmptyQueueException:
                pass
        if deadline < datetime.datetime.now():
            still_waiting = {w.worker_id for w in self.env_workers if w.waiting}
            raise TimeoutError(f"Workers {still_waiting} stuck in waiting state")
        return all_failures

    def _assert_worker_can_restart(self, worker_id: int, exception: Exception) -> None:
        """
        Checks if we can recover from an exception from a worker.
        If the restart limit is exceeded it will raise a UnityCommunicationException.
        If the exception is not recoverable it re-raises the exception.
        """
        if (
            isinstance(exception, UnityCommunicationException)
            or isinstance(exception, UnityTimeOutException)
            or isinstance(exception, UnityEnvironmentException)
            or isinstance(exception, UnityCommunicatorStoppedException)
        ):
            if self._worker_has_restart_quota(worker_id):
                return
            else:
                logger.error(

View on GitHub (pinned to 3ecb446f75)

Solutions

1. Check the Unity side for a hung environment (infinite loop in an Agent/Academy callback, blocking scene script)
2. Upgrade the ml-agents package — worker hang handling and draining logic have been improved across releases
3. Reduce num_envs to lower IPC load and identify which worker_id is stuck
4. Ensure time_scale / frame settings do not starve the environment process

When it happens

Trigger: Thrown at ml-agents/mlagents/trainers/subprocess_env_manager.py:352 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of Unity-Technologies/ml-agents@3ecb446f75 (2026-09-02). Data as JSON: /api/errors/c3166411ca8c0c95. Report an issue: GitHub.