hiyouga/LlamaFactory · error · ValueError

The number of devices in the Ray cluster ({total_devices}) s

Error message

The number of devices in the Ray cluster ({total_devices}) should be greater than num_workers ({num_workers}).

What it means

In the Ray launcher (src/llamafactory/train/tuner.py:324), after ray.init the code reads the cluster's total accelerator count (`ray.cluster_resources()[get_device_name()]`, e.g. 'GPU') and compares it to `num_workers`. If num_workers exceeds available devices, the distributed job cannot place one process per GPU and fails fast with this ValueError.

Source

Thrown at src/llamafactory/train/tuner.py:324

def _ray_training_function(ray_args: "RayArguments", config: dict[str, Any]) -> None:
    num_workers = ray_args.ray_num_workers
    master_addr = ray_args.master_addr
    master_port = ray_args.master_port
    logger.info(f"Using ray.remote mode with {num_workers} workers for distributed training.")

    # initialize ray
    if not ray.is_initialized():
        if ray_args.ray_init_kwargs is not None:
            ray.init(**ray_args.ray_init_kwargs)
        else:
            ray.init()

    # verify resources
    device_name = get_device_name().upper()
    total_devices = int(ray.cluster_resources().get(device_name, 0))
    if num_workers > total_devices:
        raise ValueError(
            f"The number of devices in the Ray cluster ({total_devices}) should be greater than num_workers ({num_workers})."
        )

    # verify master_addr
    if master_addr is None:
        master_addr = get_ray_head_node_ip()
        logger.info(f"`master_addr` is not specified, using head node ip: {master_addr}.")
    else:
        nodes = [node["NodeManagerAddress"] for node in ray.nodes() if node["Alive"]]
        if master_addr not in nodes:
            raise ValueError(f"The `master_addr` ({master_addr}) is not in Ray cluster or not alive ")

    # create placementgroup for resource management
    pg, bundle = get_placement_group(total_devices)
    ray.get(pg.ready())
    logger.info(f"Create placement group with {num_workers} bundles: {bundle}")

    # get sorted_bundle_indices

View on GitHub (pinned to f28afaf635)

Solutions

1. Reduce `num_workers` (ray_num_workers) to at most the cluster's GPU count.
2. Scale the Ray cluster: start more GPU nodes or free GPUs held by other jobs (`ray status` to inspect).
3. Check CUDA_VISIBLE_DEVICES on head/worker nodes so Ray can see all intended GPUs.
4. For multi-node, verify num_workers equals total GPUs across nodes, not per node.

Example fix

# before (yaml)
use_ray: true
ray_num_workers: 8   # cluster has 4 GPUs

# after
use_ray: true
ray_num_workers: 4

import ray
def workers_fit_cluster(num_workers: int) -> bool:
    device = "GPU"  # or derive from llamafactory get_device_name().upper()
    return num_workers <= int(ray.cluster_resources().get(device, 0))

When it happens

Trigger: `use_ray: true` with `num_workers: N` where N > GPUs visible to the Ray cluster: too few nodes/GPUs started, CUDA_VISIBLE_DEVICES restricting visibility, or workers sized for old hardware.

Common situations: Scaling configs across environments (8-GPU config on a 4-GPU box), Ray cluster where workers joined without GPUs, or GPU resources exhausted by other actors so cluster_resources reports fewer.

Related errors

• `tensor_model_parallel_size` must be >= 1.
• `pipeline_model_parallel_size` must be >= 1.
• Total Megatron Bridge parallel size ({parallel_size}) exceed
• Total Megatron Bridge parallel size ({parallel_size}) must d
• Megatron Bridge cannot be used together with MCA or HyperPar

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/0df08e9428b62284. Report an issue: GitHub.