{"record":{"id":"d9068481d278815a","repo":"2noise/ChatTTS","slug":"ray-does-not-allocate-any-gpus-on-the-driver-node","errorCode":null,"errorMessage":"Ray does not allocate any GPUs on the driver node. Consider adjusting the Ray placement group or running the driver on a GPU node.","messagePattern":"Ray does not allocate any GPUs on the driver node\\. Consider adjusting the Ray placement group or running the driver on a GPU node\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"ChatTTS/model/velocity/llm_engine.py","lineNumber":185,"sourceCode":"                placement_group_bundle_index=bundle_id,\n            )\n            worker = ray.remote(\n                num_cpus=0,\n                num_gpus=num_gpus,\n                scheduling_strategy=scheduling_strategy,\n                **ray_remote_kwargs,\n            )(RayWorkerVllm).remote(self.model_config.trust_remote_code)\n\n            worker_ip = ray.get(worker.get_node_ip.remote())\n            if worker_ip == driver_ip and self.driver_dummy_worker is None:\n                # If the worker is on the same node as the driver, we use it\n                # as the resource holder for the driver process.\n                self.driver_dummy_worker = worker\n            else:\n                self.workers.append(worker)\n\n        if self.driver_dummy_worker is None:\n            raise ValueError(\n                \"Ray does not allocate any GPUs on the driver node. Consider \"\n                \"adjusting the Ray placement group or running the driver on a \"\n                \"GPU node.\"\n            )\n\n        driver_node_id, driver_gpu_ids = ray.get(\n            self.driver_dummy_worker.get_node_and_gpu_ids.remote()\n        )\n        worker_node_and_gpu_ids = ray.get(\n            [worker.get_node_and_gpu_ids.remote() for worker in self.workers]\n        )\n\n        node_workers = defaultdict(list)\n        node_gpus = defaultdict(list)\n\n        node_workers[driver_node_id].append(0)\n        node_gpus[driver_node_id].extend(driver_gpu_ids)\n        for i, (node_id, gpu_ids) in enumerate(worker_node_and_gpu_ids, start=1):","sourceCodeStart":167,"sourceCodeEnd":203,"githubUrl":"https://github.com/2noise/ChatTTS/blob/77b89ee281cd479f5b1a787ada330dc975ca1f2a/ChatTTS/model/velocity/llm_engine.py#L167-L203","documentation":"When distributed execution uses Ray, the driver process must itself hold at least one GPU (a 'dummy worker' is created on the driver node to own GPU resources). If the Ray placement group schedules all GPU bundles onto other nodes, driver_dummy_worker stays None and engine init aborts. This is a scheduling/topology error, not a code error.","triggerScenarios":"LLM(..., distributed_executor_backend='ray') on a Ray cluster where the driver node has no GPUs or its GPUs are fully occupied by other placement groups; a placement group with fewer GPU bundles than driver+workers need.","commonSituations":"Running the driver on a CPU-only head node of a Ray cluster; GPU nodes already saturated by other jobs so the driver's bundle lands elsewhere; Ray cluster started with num-gpus=0 on the head.","solutions":["Run the driver on a GPU node (e.g. ray start on a GPU machine, then ray.init(address='auto') from there).","Free GPU capacity on the driver node or increase the placement group's GPU bundle count so one bundle fits on the driver node.","For single-node multi-GPU, prefer the default multiprocessing backend instead of Ray - this code path is then skipped entirely."],"exampleFix":"# before (driver on CPU head node)\nray.init(address='auto')\nengine = LLM(model=path, tensor_parallel_size=4, distributed_executor_backend='ray')\n\n# after (start driver where GPUs are)\n# ray start --head --num-gpus=8  on the GPU node, then:\nray.init(address='auto')\nengine = LLM(model=path, tensor_parallel_size=4, distributed_executor_backend='ray')","handlingStrategy":"validation","validationCode":"import ray\n\ndef driver_has_gpu():\n    nodes = [n for n in ray.nodes() if n.get('Alive')]\n    return any(n['Resources'].get('GPU', 0) >= 1 for n in nodes)","typeGuard":null,"tryCatchPattern":"try:\n    engine = LLM(model=path, distributed_executor_backend='ray', tensor_parallel_size=N)\nexcept ValueError as e:\n    if 'Ray does not allocate any GPUs' in str(e):\n        engine = LLM(model=path, tensor_parallel_size=N)  # fall back to multiprocessing backend\n    else:\n        raise","preventionTips":["On single-node multi-GPU, use the default multiprocessing backend, not Ray.","Ensure the driver launches from a GPU node with free GPUs before ray.init()."],"tags":["ray","distributed","gpu-scheduling","placement-group"],"backgroundTag":"gpu-resource-unavailable","analyzedSha":"77b89ee281cd479f5b1a787ada330dc975ca1f2a","analyzedAt":"2026-08-26T17:48:24.233Z","schemaVersion":2},"datasetVersion":"2026-08-26T21:11:00.512Z"}