sgl-project/sglang · error · ValueError

No IB devices configured for GPU {gpu_id}. Available GPUs: {

Error message

No IB devices configured for GPU {gpu_id}. Available GPUs: {list(parsed_config.keys())}

What it means

MooncakeTransferEngine.__init__ resolves the IB device for its GPU via get_ib_devices_for_gpu, but the parsed mapping (dict form) has no entry for this gpu_id. The message lists the GPU ids that ARE configured. Note: a plain-string config returns for any GPU; only dict configs can miss a key.

Source

Thrown at python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py:103

    Args:
        ib_device_str: The original IB device string or path to JSON file
        gpu_id: The GPU ID to get devices for

    Returns:
        IB devices string for the GPU, or None if not available
    """
    parsed_config = parse_ib_device_config(ib_device_str)
    if parsed_config is None:
        return None

    if isinstance(parsed_config, str):
        return parsed_config

    if gpu_id in parsed_config:
        return parsed_config[gpu_id]

    raise ValueError(
        f"No IB devices configured for GPU {gpu_id}. "
        f"Available GPUs: {list(parsed_config.keys())}"
    )


class MooncakeTransferEngine:
    """Shared Mooncake transfer engine for RDMA/transfer operations."""

    def __init__(
        self,
        hostname: str,
        gpu_id: Optional[int] = None,
        ib_device: Optional[str] = None,
    ):
        try:
            from mooncake.engine import TransferEngine
        except ImportError as e:
            raise ImportError(

View on GitHub (pinned to 0132848349)

Solutions

1. Add the missing GPU id (listed as 'Available GPUs' shows which keys exist) to the mapping — ensure every local GPU id on every node has an entry
2. If CUDA_VISIBLE_DEVICES remaps devices, key the mapping by the LOCAL (post-remap) gpu id or clear the remapping
3. Alternatively use a single IB device string (non-dict config), which applies to all GPUs and cannot miss

Example fix

# before
{"0": "mlx5_0", "1": "mlx5_1"}   # GPUs 2-3 -> ValueError

# after
{"0": "mlx5_0", "1": "mlx5_1", "2": "mlx5_2", "3": "mlx5_3"}

import torch

def assert_gpu_covered(cfg: str) -> None:
    parsed = parse_ib_device_config(cfg)
    if isinstance(parsed, dict):
        local_ids = range(torch.cuda.device_count())
        missing = [g for g in local_ids if g not in parsed]
        assert not missing, f"IB mapping missing local GPU ids: {missing}; configured: {list(parsed)}"

try:
    dev = get_ib_devices_for_gpu(cfg, gpu_id)
except ValueError as e:
    if "No IB devices configured for GPU" in str(e):
        raise ConfigError(f"add gpu {gpu_id} to the IB mapping") from e
    raise

When it happens

Trigger: Running with a GPU-id->IB-device JSON mapping that omits the current rank's gpu_id — e.g. mapping only GPUs 0-3 while an 8-GPU node launches ranks on GPUs 4-7, or device-id remapping via CUDA_VISIBLE_DEVICES shifting the local gpu_id.

Common situations: 8-GPU node provisioned with a 4-GPU mapping file, per-node mappings copied to the wrong host, or CUDA_VISIBLE_DEVICES changing the per-process local GPU id so it no longer matches the mapping key.

Related errors

• File {normalized_input} does not exist.
• Failed to parse JSON content from file {normalized_input}
• Failed to read JSON file {normalized_input}: {exc}
• Invalid JSON mapping: {normalized_input}
• Invalid format: expected a mapping from GPU id to IB device

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/962f5394bb39ae43. Report an issue: GitHub.