{"record":{"id":"962f5394bb39ae43","repo":"sgl-project/sglang","slug":"no-ib-devices-configured-for-gpu-gpu-id-availab","errorCode":null,"errorMessage":"No IB devices configured for GPU {gpu_id}. Available GPUs: {list(parsed_config.keys())}","messagePattern":"No IB devices configured for GPU (.+?)\\. Available GPUs: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py","lineNumber":103,"sourceCode":"\n    Args:\n        ib_device_str: The original IB device string or path to JSON file\n        gpu_id: The GPU ID to get devices for\n\n    Returns:\n        IB devices string for the GPU, or None if not available\n    \"\"\"\n    parsed_config = parse_ib_device_config(ib_device_str)\n    if parsed_config is None:\n        return None\n\n    if isinstance(parsed_config, str):\n        return parsed_config\n\n    if gpu_id in parsed_config:\n        return parsed_config[gpu_id]\n\n    raise ValueError(\n        f\"No IB devices configured for GPU {gpu_id}. \"\n        f\"Available GPUs: {list(parsed_config.keys())}\"\n    )\n\n\nclass MooncakeTransferEngine:\n    \"\"\"Shared Mooncake transfer engine for RDMA/transfer operations.\"\"\"\n\n    def __init__(\n        self,\n        hostname: str,\n        gpu_id: Optional[int] = None,\n        ib_device: Optional[str] = None,\n    ):\n        try:\n            from mooncake.engine import TransferEngine\n        except ImportError as e:\n            raise ImportError(","sourceCodeStart":85,"sourceCodeEnd":121,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py#L85-L121","documentation":"MooncakeTransferEngine.__init__ resolves the IB device for its GPU via get_ib_devices_for_gpu, but the parsed mapping (dict form) has no entry for this gpu_id. The message lists the GPU ids that ARE configured. Note: a plain-string config returns for any GPU; only dict configs can miss a key.","triggerScenarios":"Running with a GPU-id->IB-device JSON mapping that omits the current rank's gpu_id — e.g. mapping only GPUs 0-3 while an 8-GPU node launches ranks on GPUs 4-7, or device-id remapping via CUDA_VISIBLE_DEVICES shifting the local gpu_id.","commonSituations":"8-GPU node provisioned with a 4-GPU mapping file, per-node mappings copied to the wrong host, or CUDA_VISIBLE_DEVICES changing the per-process local GPU id so it no longer matches the mapping key.","solutions":["Add the missing GPU id (listed as 'Available GPUs' shows which keys exist) to the mapping — ensure every local GPU id on every node has an entry","If CUDA_VISIBLE_DEVICES remaps devices, key the mapping by the LOCAL (post-remap) gpu id or clear the remapping","Alternatively use a single IB device string (non-dict config), which applies to all GPUs and cannot miss"],"exampleFix":"# before\n{\"0\": \"mlx5_0\", \"1\": \"mlx5_1\"}   # GPUs 2-3 -> ValueError\n\n# after\n{\"0\": \"mlx5_0\", \"1\": \"mlx5_1\", \"2\": \"mlx5_2\", \"3\": \"mlx5_3\"}","handlingStrategy":"validation","validationCode":"import torch\n\ndef assert_gpu_covered(cfg: str) -> None:\n    parsed = parse_ib_device_config(cfg)\n    if isinstance(parsed, dict):\n        local_ids = range(torch.cuda.device_count())\n        missing = [g for g in local_ids if g not in parsed]\n        assert not missing, f\"IB mapping missing local GPU ids: {missing}; configured: {list(parsed)}\"","typeGuard":null,"tryCatchPattern":"try:\n    dev = get_ib_devices_for_gpu(cfg, gpu_id)\nexcept ValueError as e:\n    if \"No IB devices configured for GPU\" in str(e):\n        raise ConfigError(f\"add gpu {gpu_id} to the IB mapping\") from e\n    raise","preventionTips":["Key the mapping by post-CUDA_VISIBLE_DEVICES LOCAL gpu ids, or avoid remapping","Cover every local GPU on every node in the mapping file","Use the single-device string form when all GPUs share an IB device"],"tags":["config","mooncake","ib-devices","gpu-mapping"],"backgroundTag":"missing-config-key","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}