{"record":{"id":"0df08e9428b62284","repo":"hiyouga/LlamaFactory","slug":"the-number-of-devices-in-the-ray-cluster-total-d","errorCode":null,"errorMessage":"The number of devices in the Ray cluster ({total_devices}) should be greater than num_workers ({num_workers}).","messagePattern":"The number of devices in the Ray cluster \\((.+?)\\) should be greater than num_workers \\((.+?)\\)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/train/tuner.py","lineNumber":324,"sourceCode":"\ndef _ray_training_function(ray_args: \"RayArguments\", config: dict[str, Any]) -> None:\n    num_workers = ray_args.ray_num_workers\n    master_addr = ray_args.master_addr\n    master_port = ray_args.master_port\n    logger.info(f\"Using ray.remote mode with {num_workers} workers for distributed training.\")\n\n    # initialize ray\n    if not ray.is_initialized():\n        if ray_args.ray_init_kwargs is not None:\n            ray.init(**ray_args.ray_init_kwargs)\n        else:\n            ray.init()\n\n    # verify resources\n    device_name = get_device_name().upper()\n    total_devices = int(ray.cluster_resources().get(device_name, 0))\n    if num_workers > total_devices:\n        raise ValueError(\n            f\"The number of devices in the Ray cluster ({total_devices}) should be greater than num_workers ({num_workers}).\"\n        )\n\n    # verify master_addr\n    if master_addr is None:\n        master_addr = get_ray_head_node_ip()\n        logger.info(f\"`master_addr` is not specified, using head node ip: {master_addr}.\")\n    else:\n        nodes = [node[\"NodeManagerAddress\"] for node in ray.nodes() if node[\"Alive\"]]\n        if master_addr not in nodes:\n            raise ValueError(f\"The `master_addr` ({master_addr}) is not in Ray cluster or not alive \")\n\n    # create placementgroup for resource management\n    pg, bundle = get_placement_group(total_devices)\n    ray.get(pg.ready())\n    logger.info(f\"Create placement group with {num_workers} bundles: {bundle}\")\n\n    # get sorted_bundle_indices","sourceCodeStart":306,"sourceCodeEnd":342,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/train/tuner.py#L306-L342","documentation":"In the Ray launcher (src/llamafactory/train/tuner.py:324), after ray.init the code reads the cluster's total accelerator count (`ray.cluster_resources()[get_device_name()]`, e.g. 'GPU') and compares it to `num_workers`. If num_workers exceeds available devices, the distributed job cannot place one process per GPU and fails fast with this ValueError.","triggerScenarios":"`use_ray: true` with `num_workers: N` where N > GPUs visible to the Ray cluster: too few nodes/GPUs started, CUDA_VISIBLE_DEVICES restricting visibility, or workers sized for old hardware.","commonSituations":"Scaling configs across environments (8-GPU config on a 4-GPU box), Ray cluster where workers joined without GPUs, or GPU resources exhausted by other actors so cluster_resources reports fewer.","solutions":["Reduce `num_workers` (ray_num_workers) to at most the cluster's GPU count.","Scale the Ray cluster: start more GPU nodes or free GPUs held by other jobs (`ray status` to inspect).","Check CUDA_VISIBLE_DEVICES on head/worker nodes so Ray can see all intended GPUs.","For multi-node, verify num_workers equals total GPUs across nodes, not per node."],"exampleFix":"# before (yaml)\nuse_ray: true\nray_num_workers: 8   # cluster has 4 GPUs\n\n# after\nuse_ray: true\nray_num_workers: 4","handlingStrategy":"validation","validationCode":"import ray\ndef workers_fit_cluster(num_workers: int) -> bool:\n    device = \"GPU\"  # or derive from llamafactory get_device_name().upper()\n    return num_workers <= int(ray.cluster_resources().get(device, 0))","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Run `ray status` and compare GPU count to ray_num_workers before submitting.","Automate the cluster_resources vs num_workers check in job-launch scripts.","Set CUDA_VISIBLE_DEVICES explicitly on Ray head and worker nodes."],"tags":["ray","distributed","gpu","resource-planning"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}