{"record":{"id":"c3166411ca8c0c95","repo":"Unity-Technologies/ml-agents","slug":"workers-still-waiting-stuck-in-waiting-state","errorCode":null,"errorMessage":"Workers {still_waiting} stuck in waiting state","messagePattern":"Workers (.+?) stuck in waiting state","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"ml-agents/mlagents/trainers/subprocess_env_manager.py","lineNumber":352,"sourceCode":"        \"\"\"\n        all_failures = {}\n        workers_still_pending = {w.worker_id for w in self.env_workers if w.waiting}\n        deadline = datetime.datetime.now() + datetime.timedelta(minutes=1)\n        while workers_still_pending and deadline > datetime.datetime.now():\n            try:\n                while True:\n                    step: EnvironmentResponse = self.step_queue.get_nowait()\n                    if step.cmd == EnvironmentCommand.ENV_EXITED:\n                        workers_still_pending.add(step.worker_id)\n                        all_failures[step.worker_id] = step.payload\n                    else:\n                        workers_still_pending.remove(step.worker_id)\n                        self.env_workers[step.worker_id].waiting = False\n            except EmptyQueueException:\n                pass\n        if deadline < datetime.datetime.now():\n            still_waiting = {w.worker_id for w in self.env_workers if w.waiting}\n            raise TimeoutError(f\"Workers {still_waiting} stuck in waiting state\")\n        return all_failures\n\n    def _assert_worker_can_restart(self, worker_id: int, exception: Exception) -> None:\n        \"\"\"\n        Checks if we can recover from an exception from a worker.\n        If the restart limit is exceeded it will raise a UnityCommunicationException.\n        If the exception is not recoverable it re-raises the exception.\n        \"\"\"\n        if (\n            isinstance(exception, UnityCommunicationException)\n            or isinstance(exception, UnityTimeOutException)\n            or isinstance(exception, UnityEnvironmentException)\n            or isinstance(exception, UnityCommunicatorStoppedException)\n        ):\n            if self._worker_has_restart_quota(worker_id):\n                return\n            else:\n                logger.error(","sourceCodeStart":334,"sourceCodeEnd":370,"githubUrl":"https://github.com/Unity-Technologies/ml-agents/blob/3ecb446f75d1e7400eb404c562dc005d3164cffc/ml-agents/mlagents/trainers/subprocess_env_manager.py#L334-L370","documentation":"A sentinel error raised by SubprocessEnvManager._drain_step_queue (invoked from _restart_failed_workers) when, after waiting up to a one-minute deadline, some environment workers are still flagged as 'waiting' — i.e. they were sent a step/reset command but never produced a response on the step queue. It means those worker processes or their Unity environments hung or died without sending ENV_EXITED, so the manager cannot cleanly collect their state before restarting them.","triggerScenarios":"Thrown at ml-agents/mlagents/trainers/subprocess_env_manager.py:352 when the library encounters an invalid state.","commonSituations":"See trigger scenarios.","solutions":["Check the Unity side for a hung environment (infinite loop in an Agent/Academy callback, blocking scene script)","Upgrade the ml-agents package — worker hang handling and draining logic have been improved across releases","Reduce num_envs to lower IPC load and identify which worker_id is stuck","Ensure time_scale / frame settings do not starve the environment process"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"3ecb446f75d1e7400eb404c562dc005d3164cffc","analyzedAt":"2026-09-02T16:33:12.832Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T21:17:11.164Z"}