hiyouga/LlamaFactory · error · ValueError

The `master_addr` ({master_addr}) is not in Ray cluster or n

Error message

The `master_addr` ({master_addr}) is not in Ray cluster or not alive

What it means

LlamaFactory raises this ValueError when launching distributed training with the Ray backend and the explicitly supplied `master_addr` does not match any alive node in the Ray cluster. The master address is used to anchor the torch distributed rendezvous, so it must be the address of a live Ray node (normally the head node). The check compares the given value against `node['NodeManagerAddress']` for every node where `Alive` is true.

Source

Thrown at src/llamafactory/train/tuner.py:335

        else:
            ray.init()

    # verify resources
    device_name = get_device_name().upper()
    total_devices = int(ray.cluster_resources().get(device_name, 0))
    if num_workers > total_devices:
        raise ValueError(
            f"The number of devices in the Ray cluster ({total_devices}) should be greater than num_workers ({num_workers})."
        )

    # verify master_addr
    if master_addr is None:
        master_addr = get_ray_head_node_ip()
        logger.info(f"`master_addr` is not specified, using head node ip: {master_addr}.")
    else:
        nodes = [node["NodeManagerAddress"] for node in ray.nodes() if node["Alive"]]
        if master_addr not in nodes:
            raise ValueError(f"The `master_addr` ({master_addr}) is not in Ray cluster or not alive ")

    # create placementgroup for resource management
    pg, bundle = get_placement_group(total_devices)
    ray.get(pg.ready())
    logger.info(f"Create placement group with {num_workers} bundles: {bundle}")

    # get sorted_bundle_indices
    sorted_bundle_indices = sort_placement_group_by_node_ip(pg, master_addr)

    # get master port
    if master_port is None:
        master_port = find_available_port()
        logger.info(f"`master_port` is not specified, using available port: {master_port}.")
    master_port = str(master_port)

    # backing up environment variables
    current_env = dict(os.environ.items())

View on GitHub (pinned to f28afaf635)

Solutions

Remove `master_addr` from the YAML/CLI so LlamaFactory auto-selects the head node IP via `get_ray_head_node_ip()`
Run `ray status` / `ray.nodes()` (or `python -c "import ray; ray.init(); print([n['NodeManagerAddress'] for n in ray.nodes() if n['Alive']])"`) and set `master_addr` to one of the printed addresses exactly
If the intended node is missing from the list, restart it with `ray start` and wait for it to register as Alive before retrying
If addresses look wrong, restart the whole cluster (`ray stop && ray start --head ...`) to refresh NodeManagerAddress values

Example fix

# before (yaml)
ray: true
master_addr: 10.0.0.42  # stale IP from an old cluster

# after (yaml)
ray: true
# master_addr omitted; head node IP is auto-detected and logged

Defensive patterns

Strategy: validation

Validate before calling

import ray

def validate_master_addr(master_addr: str | None) -> str:
    if master_addr is None:
        return ray.get_runtime_context().get_address_info  # or let llamafactory pick head ip
    alive = [n["NodeManagerAddress"] for n in ray.nodes() if n["Alive"]]
    if master_addr not in alive:
        raise SystemExit(f"master_addr {master_addr} not in alive Ray nodes {alive}; rerun with a listed address or omit it")
    return master_addr

Try / catch

try:
    run_exp(args)  # ray launch path
except ValueError as e:
    if "master_addr" in str(e):
        logger.error("Ray master_addr invalid; retrying with auto head-node IP")
        args.master_addr = None  # next attempt auto-selects
    else:
        raise

Prevention

Omit master_addr in Ray configs unless you manage the cluster yourself
Script cluster startup + config generation together so addresses never go stale
Log `ray.nodes()` alive addresses right before launch in CI/nightly jobs

When it happens

Trigger: Calling `tuner.py`'s Ray launch path (e.g. `ray_start` in run_exp with `use_ray: true`) with a `master_addr` that is (a) a hostname/IP different from the Ray NodeManagerAddress strings, (b) a node that has died or not yet registered, or (c) copied from a previous cluster that no longer exists.

Common situations: Hardcoding an old head-node IP after the Ray cluster was restarted; using a public/external IP while Ray reports internal addresses; a worker node dying between cluster startup and training launch; mixing autodetected and manual address values in a config template.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/fef9cbc6b2a624bf. Report an issue: GitHub.