{"record":{"id":"d0b70ffc7f7fe992","repo":"vllm-project/vllm","slug":"worker-has-been-garbage-collected","errorCode":null,"errorMessage":"Worker has been garbage collected","messagePattern":"Worker has been garbage collected","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"vllm/distributed/elastic_ep/elastic_execute.py","lineNumber":161,"sourceCode":"\n    return physical_to_logical, num_local_physical_experts, num_logical_experts\n\n\nclass ElasticEPScalingExecutor:\n    def __init__(self, worker):\n        self.worker_ref = weakref.ref(worker)\n        self.reconfig_request = None\n        self._staged_moe_quant_methods: dict[nn.Module, FusedMoEMethodBase] = {}\n        self._async_executor = ThreadPoolExecutor(\n            max_workers=1, thread_name_prefix=\"ElasticEPAsync\"\n        )\n        self._async_future: Future[None] | None = None\n\n    @property\n    def worker(self):\n        worker = self.worker_ref()\n        if worker is None:\n            raise RuntimeError(\"Worker has been garbage collected\")\n        return worker\n\n    def execute(self, execute_method: str, *args, **kwargs):\n        method = getattr(self, execute_method, None)\n        if method is None:\n            raise ValueError(f\"Unknown execute method: {execute_method}\")\n        return method(*args, **kwargs)\n\n    def start_async(self, execute_method: str, *args, **kwargs) -> str:\n        if self._async_future is not None:\n            raise RuntimeError(\"Another Elastic EP async method is active\")\n        if args and isinstance(args[0], ReconfigureDistributedRequest):\n            self.reconfig_request = args[0]\n        dp_rank = self.worker.vllm_config.parallel_config.data_parallel_rank\n        done_key = f\"eep_async/{execute_method}/{dp_rank}/{self.worker.rank}\"\n        self._async_future = self._async_executor.submit(\n            self._run_async, execute_method, *args, **kwargs\n        )","sourceCodeStart":143,"sourceCodeEnd":179,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/elastic_ep/elastic_execute.py#L143-L179","documentation":"ElasticEPScalingExecutor holds its worker via weakref.ref(worker) so the executor never keeps the model worker alive. The `worker` property dereferences the ref; if the referent has been garbage collected (no strong references remain anywhere), it raises RuntimeError — a use-after-free guard for the async reconfiguration thread accessing a dead worker.","triggerScenarios":"Creating ElasticEPScalingExecutor(worker) while all other strong references to the worker are dropped (e.g. the constructing scope returns and only the executor survives), then calling execute()/start_async() which touches self.worker; the worker being deliberately torn down during engine shutdown while an async reconfigure is still pending.","commonSituations":"Engine shutdown races where the async thread pool task outlives the worker; test code constructing the executor with a temporary worker object; refactors that accidentally store the executor globally but not the worker.","solutions":["Keep a strong reference to the worker for the executor's lifetime (the owner that creates the executor should retain the worker)","On shutdown, wait for/join the async future (executor._async_future / shutdown of the ThreadPoolExecutor) before dropping worker references","In tests, assign the worker to a long-lived variable and shut the executor down before teardown"],"exampleFix":"# before\nexecutor = ElasticEPScalingExecutor(make_worker())  # temp worker\nexecutor.start_async(\"reconfigure\", req)  # later: RuntimeError\n\n# after\nworker = make_worker()  # strong reference kept by owner\nexecutor = ElasticEPScalingExecutor(worker)\n...\nexecutor.shutdown()  # join async work before worker goes out of scope","handlingStrategy":"validation","validationCode":"worker = executor.worker_ref()\nif worker is None:\n    raise RuntimeError(\"worker collected; abort before scheduling async work\")","typeGuard":"def executor_alive(executor: \"ElasticEPScalingExecutor\") -> bool:\n    return executor.worker_ref() is not None","tryCatchPattern":"try:\n    worker = executor.worker\nexcept RuntimeError:\n    # worker gone: cancel pending async work instead of using it\n    executor._async_executor.shutdown(wait=False, cancel_futures=True)\n    raise","preventionTips":["Retain a strong reference to the worker wherever the executor lives","Join the executor's async future before releasing the worker on shutdown","Avoid constructing the executor with temporaries in tests"],"tags":["elastic-ep","lifetime-management","garbage-collection","race-condition"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}