{"record":{"id":"fef9cbc6b2a624bf","repo":"hiyouga/LlamaFactory","slug":"the-master-addr-master-addr-is-not-in-ray-cl","errorCode":null,"errorMessage":"The `master_addr` ({master_addr}) is not in Ray cluster or not alive ","messagePattern":"The `master_addr` \\((.+?)\\) is not in Ray cluster or not alive ","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/train/tuner.py","lineNumber":335,"sourceCode":"        else:\n            ray.init()\n\n    # verify resources\n    device_name = get_device_name().upper()\n    total_devices = int(ray.cluster_resources().get(device_name, 0))\n    if num_workers > total_devices:\n        raise ValueError(\n            f\"The number of devices in the Ray cluster ({total_devices}) should be greater than num_workers ({num_workers}).\"\n        )\n\n    # verify master_addr\n    if master_addr is None:\n        master_addr = get_ray_head_node_ip()\n        logger.info(f\"`master_addr` is not specified, using head node ip: {master_addr}.\")\n    else:\n        nodes = [node[\"NodeManagerAddress\"] for node in ray.nodes() if node[\"Alive\"]]\n        if master_addr not in nodes:\n            raise ValueError(f\"The `master_addr` ({master_addr}) is not in Ray cluster or not alive \")\n\n    # create placementgroup for resource management\n    pg, bundle = get_placement_group(total_devices)\n    ray.get(pg.ready())\n    logger.info(f\"Create placement group with {num_workers} bundles: {bundle}\")\n\n    # get sorted_bundle_indices\n    sorted_bundle_indices = sort_placement_group_by_node_ip(pg, master_addr)\n\n    # get master port\n    if master_port is None:\n        master_port = find_available_port()\n        logger.info(f\"`master_port` is not specified, using available port: {master_port}.\")\n    master_port = str(master_port)\n\n    # backing up environment variables\n    current_env = dict(os.environ.items())\n","sourceCodeStart":317,"sourceCodeEnd":353,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/train/tuner.py#L317-L353","documentation":"LlamaFactory raises this ValueError when launching distributed training with the Ray backend and the explicitly supplied `master_addr` does not match any alive node in the Ray cluster. The master address is used to anchor the torch distributed rendezvous, so it must be the address of a live Ray node (normally the head node). The check compares the given value against `node['NodeManagerAddress']` for every node where `Alive` is true.","triggerScenarios":"Calling `tuner.py`'s Ray launch path (e.g. `ray_start` in run_exp with `use_ray: true`) with a `master_addr` that is (a) a hostname/IP different from the Ray NodeManagerAddress strings, (b) a node that has died or not yet registered, or (c) copied from a previous cluster that no longer exists.","commonSituations":"Hardcoding an old head-node IP after the Ray cluster was restarted; using a public/external IP while Ray reports internal addresses; a worker node dying between cluster startup and training launch; mixing autodetected and manual address values in a config template.","solutions":["Remove `master_addr` from the YAML/CLI so LlamaFactory auto-selects the head node IP via `get_ray_head_node_ip()`","Run `ray status` / `ray.nodes()` (or `python -c \"import ray; ray.init(); print([n['NodeManagerAddress'] for n in ray.nodes() if n['Alive']])\"`) and set `master_addr` to one of the printed addresses exactly","If the intended node is missing from the list, restart it with `ray start` and wait for it to register as Alive before retrying","If addresses look wrong, restart the whole cluster (`ray stop && ray start --head ...`) to refresh NodeManagerAddress values"],"exampleFix":"# before (yaml)\nray: true\nmaster_addr: 10.0.0.42  # stale IP from an old cluster\n\n# after (yaml)\nray: true\n# master_addr omitted; head node IP is auto-detected and logged","handlingStrategy":"validation","validationCode":"import ray\n\ndef validate_master_addr(master_addr: str | None) -> str:\n    if master_addr is None:\n        return ray.get_runtime_context().get_address_info  # or let llamafactory pick head ip\n    alive = [n[\"NodeManagerAddress\"] for n in ray.nodes() if n[\"Alive\"]]\n    if master_addr not in alive:\n        raise SystemExit(f\"master_addr {master_addr} not in alive Ray nodes {alive}; rerun with a listed address or omit it\")\n    return master_addr","typeGuard":null,"tryCatchPattern":"try:\n    run_exp(args)  # ray launch path\nexcept ValueError as e:\n    if \"master_addr\" in str(e):\n        logger.error(\"Ray master_addr invalid; retrying with auto head-node IP\")\n        args.master_addr = None  # next attempt auto-selects\n    else:\n        raise","preventionTips":["Omit master_addr in Ray configs unless you manage the cluster yourself","Script cluster startup + config generation together so addresses never go stale","Log `ray.nodes()` alive addresses right before launch in CI/nightly jobs"],"tags":["ray","distributed-training","configuration","network"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}