2noise/ChatTTS · critical · ValueError

Ray does not allocate any GPUs on the driver node. Consider

Error message

Ray does not allocate any GPUs on the driver node. Consider adjusting the Ray placement group or running the driver on a GPU node.

What it means

When distributed execution uses Ray, the driver process must itself hold at least one GPU (a 'dummy worker' is created on the driver node to own GPU resources). If the Ray placement group schedules all GPU bundles onto other nodes, driver_dummy_worker stays None and engine init aborts. This is a scheduling/topology error, not a code error.

Source

Thrown at ChatTTS/model/velocity/llm_engine.py:185

                placement_group_bundle_index=bundle_id,
            )
            worker = ray.remote(
                num_cpus=0,
                num_gpus=num_gpus,
                scheduling_strategy=scheduling_strategy,
                **ray_remote_kwargs,
            )(RayWorkerVllm).remote(self.model_config.trust_remote_code)

            worker_ip = ray.get(worker.get_node_ip.remote())
            if worker_ip == driver_ip and self.driver_dummy_worker is None:
                # If the worker is on the same node as the driver, we use it
                # as the resource holder for the driver process.
                self.driver_dummy_worker = worker
            else:
                self.workers.append(worker)

        if self.driver_dummy_worker is None:
            raise ValueError(
                "Ray does not allocate any GPUs on the driver node. Consider "
                "adjusting the Ray placement group or running the driver on a "
                "GPU node."
            )

        driver_node_id, driver_gpu_ids = ray.get(
            self.driver_dummy_worker.get_node_and_gpu_ids.remote()
        )
        worker_node_and_gpu_ids = ray.get(
            [worker.get_node_and_gpu_ids.remote() for worker in self.workers]
        )

        node_workers = defaultdict(list)
        node_gpus = defaultdict(list)

        node_workers[driver_node_id].append(0)
        node_gpus[driver_node_id].extend(driver_gpu_ids)
        for i, (node_id, gpu_ids) in enumerate(worker_node_and_gpu_ids, start=1):

View on GitHub (pinned to 77b89ee281)

Solutions

1. Run the driver on a GPU node (e.g. ray start on a GPU machine, then ray.init(address='auto') from there).
2. Free GPU capacity on the driver node or increase the placement group's GPU bundle count so one bundle fits on the driver node.
3. For single-node multi-GPU, prefer the default multiprocessing backend instead of Ray - this code path is then skipped entirely.

Example fix

# before (driver on CPU head node)
ray.init(address='auto')
engine = LLM(model=path, tensor_parallel_size=4, distributed_executor_backend='ray')

# after (start driver where GPUs are)
# ray start --head --num-gpus=8  on the GPU node, then:
ray.init(address='auto')
engine = LLM(model=path, tensor_parallel_size=4, distributed_executor_backend='ray')

import ray

def driver_has_gpu():
    nodes = [n for n in ray.nodes() if n.get('Alive')]
    return any(n['Resources'].get('GPU', 0) >= 1 for n in nodes)

try:
    engine = LLM(model=path, distributed_executor_backend='ray', tensor_parallel_size=N)
except ValueError as e:
    if 'Ray does not allocate any GPUs' in str(e):
        engine = LLM(model=path, tensor_parallel_size=N)  # fall back to multiprocessing backend
    else:
        raise

When it happens

Trigger: LLM(..., distributed_executor_backend='ray') on a Ray cluster where the driver node has no GPUs or its GPUs are fully occupied by other placement groups; a placement group with fewer GPU bundles than driver+workers need.

Common situations: Running the driver on a CPU-only head node of a Ray cluster; GPU nodes already saturated by other jobs so the driver's bundle lands elsewhere; Ray cluster started with num-gpus=0 on the head.

AI-assisted analysis of 2noise/ChatTTS@77b89ee281 (2026-08-26). Data as JSON: /api/errors/d9068481d278815a. Report an issue: GitHub.