sgl-project/sglang · error · ValueError

Invalid number of GPUs requested: {N} (available: {num_devic

Error message

Invalid number of GPUs requested: {N} (available: {num_devices})

What it means

After resolving a --num-gpu/--num-gpus override, multigpu_launch validates every requested GPU count N: it must be >= 2 (single-GPU makes multi-process launch pointless) and <= the number of visible CUDA devices. Otherwise it aborts before spawning workers.

Source

Thrown at python/sglang/kernels/ops/communication/mp.py:163

        if rank != 0:
            sys.stdout = open(os.devnull, "w")
        torch.cuda.set_device(rank)
        return sys.exit(inner())
    assert pid_key not in os.environ
    if name != "__main__":
        return logger.warning(
            f"{file} can not directly run with `pytest`. "
            "Use `python` to invoke it, which will internally relaunch it "
            "under torchrun for each requested number of GPUs."
        )
    num_devices = torch.cuda.device_count()
    override, forwarded_args = _extract_num_gpus_override(sys.argv[1:])
    if override is not None:
        logger.info(f"--num-gpu override: running only with {override}")
        num_gpus = override
        for N in num_gpus:
            if N <= 1 or N > num_devices:
                raise ValueError(
                    f"Invalid number of GPUs requested: {N} "
                    f"(available: {num_devices})"
                )
    os.environ[env_key] = "1"
    os.environ[pid_key] = str(os.getpid())
    os.environ.setdefault("OMP_NUM_THREADS", "1")
    os.environ.setdefault("GLOO_SOCKET_IFNAME", "lo")  # single-machine setup
    # Unbuffered child stdout: when a worker is killed on timeout, pytest's
    # block-buffered progress output is otherwise lost or flushed out of
    # order into the CI log, making it impossible to tell which test hung.
    os.environ.setdefault("PYTHONUNBUFFERED", "1")
    signal.signal(signal.SIGINT, signal.default_int_handler)
    runnable: List[int] = []
    for N in sorted(num_gpus):
        assert N > 1
        if N > num_devices:
            logger.warning(f"Skipping {kind} with {N} GPUs ({num_devices} available)")
            continue

View on GitHub (pinned to 0132848349)

Solutions

1. Request a count within [2, torch.cuda.device_count()]: check with `python -c "import torch; print(torch.cuda.device_count())"`
2. Fix CUDA_VISIBLE_DEVICES so the intended number of GPUs is visible before launching
3. Drop the --num-gpu flag to run the default all-GPU sweep

Example fix

# before
CUDA_VISIBLE_DEVICES=0 pytest test_mp.py --num-gpu=4
# after
CUDA_VISIBLE_DEVICES=0,1,2,3 pytest test_mp.py --num-gpu=4

import torch
n = torch.cuda.device_count()
assert all(1 < N <= n for N in requested), f'request {requested} vs {n} visible GPUs'

When it happens

Trigger: Running `--num-gpu 8` on a 4-GPU node, or `--num-gpu 1`/`--num-gpu 0`; also triggered when CUDA_VISIBLE_DEVICES hides devices so torch.cuda.device_count() is smaller than the requested N.

Common situations: CI boxes with fewer GPUs than the developer's machine; docker runs that map only a subset of GPUs; accidentally passing a GPU index (e.g. 3) instead of a count.

Related errors

• All inputs must be on the same device.
• All inputs must be on the same device.
• ReplaySSM inputs must be on the same device.
• {name} must be on device {device}, got {t.device}
• kv must be on q's device {device}, got {kv.device}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9b9cd2ada97c4209. Report an issue: GitHub.