{"record":{"id":"c69695b5d6f59495","repo":"microsoft/qlib","slug":"state-queue-is-already-exhausted-but-the-environm","errorCode":null,"errorMessage":"State queue is already exhausted, but the environment is still receiving action.","messagePattern":"State queue is already exhausted, but the environment is still receiving action\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"qlib/rl/utils/env_wrapper.py","lineNumber":202,"sourceCode":"            obs = self.state_interpreter(sim_state)\n\n            self.status[\"obs_history\"].append(obs)\n\n            return obs\n\n        except StopIteration:\n            # The environment should be recycled because it's in a dead state.\n            self.seed_iterator = None\n            return generate_nan_observation(self.observation_space)\n\n    def step(self, policy_action: PolicyActType, **kwargs: Any) -> Tuple[ObsType, float, bool, InfoDict]:\n        \"\"\"Environment step.\n\n        See the code along with comments to get a sequence of things happening here.\n        \"\"\"\n\n        if self.seed_iterator is None:\n            raise RuntimeError(\"State queue is already exhausted, but the environment is still receiving action.\")\n\n        # Clear the logged information from last step\n        self.logger.reset()\n\n        # Action is what we have got from policy\n        self.status[\"action_history\"].append(policy_action)\n        action = self.action_interpreter(self.simulator.get_state(), policy_action)\n\n        # This update must be after action interpreter and before simulator.\n        self.status[\"cur_step\"] += 1\n\n        # Use the converted action of update the simulator\n        self.simulator.step(action)\n\n        # Update \"done\" first, as this status might be used by reward_fn later\n        done = self.simulator.done()\n        self.status[\"done\"] = done\n","sourceCodeStart":184,"sourceCodeEnd":220,"githubUrl":"https://github.com/microsoft/qlib/blob/79633dd9506ea689e5400dea0197717b5b3d74b7/qlib/rl/utils/env_wrapper.py#L184-L220","documentation":"RuntimeError in `EnvWrapper.step` (qlib/rl/utils/env_wrapper.py:202). `step` requires an active simulator seeded by a prior successful reset; if `seed_iterator` is None (exhausted, wrapper dead) the wrapper can no longer accept actions and raises. A dead wrapper should have been recycled after the NaN observation returned by the final reset.","triggerScenarios":"Ignoring the NaN terminal observation that reset returns when the seed queue is exhausted and continuing to call `step()`; a policy loop that doesn't check the done/invalid flag on the last environment in a FiniteVectorEnv; manual stepping after the finite episode set is spent.","commonSituations":"Custom rollout code that stops only on `done` but the wrapper dies at data exhaustion before done; aggregation code that steps all envs a fixed number of times regardless of validity; races where one worker's seed iterator finishes earlier than others'.","solutions":["After each reset, check the observation validity (e.g. `finite_env.is_invalid(obs)`) and stop stepping that environment when invalid.","In vectorized settings rely on qlib's vector env to mask/recycle exhausted environments rather than stepping every slot uniformly.","Size the seed iterator to the intended number of episodes so exhaustion aligns with the end of the phase."],"exampleFix":"// before\nobs = env.reset()\nwhile True:\n    obs, rew, done, info = env.step(policy(obs))  # keeps stepping even after NaN obs\n// after\nfrom qlib.rl.utils.finite_env import is_invalid\nobs = env.reset()\nwhile not is_invalid(obs):\n    obs, rew, done, info = env.step(policy(obs))","handlingStrategy":"validation","validationCode":"from qlib.rl.utils.finite_env import is_invalid\n\ndef should_step(wrapper, obs) -> bool:\n    return wrapper.seed_iterator is not None and not is_invalid(obs)","typeGuard":"def env_step_safe(wrapper) -> bool:\n    return getattr(wrapper, \"seed_iterator\", None) is not None","tryCatchPattern":"try:\n    obs, rew, done, info = wrapper.step(action)\nexcept RuntimeError as e:\n    if \"still receiving action\" in str(e):\n        break  # environment exhausted: end rollout for this env\n    raise","preventionTips":["Check is_invalid(obs) after every reset and stop stepping invalid environments.","Use qlib's vector env masking instead of stepping all env slots uniformly.","Don't write rollout loops that stop only on done; also stop on invalid observations."],"tags":["rl","env-wrapper","lifecycle","rollout","seed-iterator"],"backgroundTag":null,"analyzedSha":"79633dd9506ea689e5400dea0197717b5b3d74b7","analyzedAt":"2026-08-15T07:01:27.511Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}