{"record":{"id":"e56019cf0ede8a04","repo":"sgl-project/sglang","slug":"mm-feature-transport-cuda-ipc-only-supports-a-si","errorCode":null,"errorMessage":"--mm-feature-transport=cuda_ipc only supports a single node.","messagePattern":"--mm-feature-transport=cuda_ipc only supports a single node\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/server_args.py","lineNumber":9010,"sourceCode":"            handle_kind = \"CUDA FABRIC\" if cfg.nnodes > 1 else \"POSIX FD\"\n            logger.info(\n                \"Using CUDA VMM for multimodal features with %s sharing: \"\n                \"reserving up to %d MiB on base GPU %d across %d tokenizer \"\n                \"worker(s). This reduces KV cache headroom; a full pool falls \"\n                \"back to inline CPU transport.\",\n                handle_kind,\n                pool_budget_mb,\n                cfg.base_gpu_id,\n                cfg.tokenizer_worker_num,\n            )\n\n        if requested_transport == \"cuda_ipc\":\n            if not is_cuda():\n                raise ValueError(\n                    \"--mm-feature-transport=cuda_ipc requires NVIDIA CUDA.\"\n                )\n            if cfg.nnodes != 1:\n                raise ValueError(\n                    \"--mm-feature-transport=cuda_ipc only supports a single node.\"\n                )\n\n            pool_budget_mb = envs.SGLANG_MM_FEATURE_CACHE_MB.get()\n            logger.info(\n                \"Using CUDA IPC for multimodal features: reserving up to %d MiB \"\n                \"on base GPU %d across %d tokenizer worker(s). This reduces KV \"\n                \"cache headroom; a full pool falls back to CPU transport.\",\n                pool_budget_mb,\n                cfg.base_gpu_id,\n                cfg.tokenizer_worker_num,\n            )\n            logger.info(\n                \"CUDA IPC pool-handle caching is %s. It reuses mappings to the \"\n                \"existing bounded pool without reserving another pool; set \"\n                \"SGLANG_USE_IPC_POOL_HANDLE_CACHE=0 to disable it.\",\n                (\n                    \"enabled\"","sourceCodeStart":8992,"sourceCodeEnd":9028,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/server_args.py#L8992-L9028","documentation":"ServerArgs validation rejects --mm-feature-transport=cuda_ipc in multi-node deployments (cfg.nnodes != 1). CUDA IPC handles are only valid between processes on the same physical node; they cannot cross network boundaries, so multimodal feature transfer over IPC is restricted to single-node launches.","triggerScenarios":"Starting a multi-node SGLang cluster (--nnodes 2 or more, or --dp ... with multiple hosts) with --mm-feature-transport cuda_ipc on any node; the nnodes check fails during arg resolution regardless of GPU vendor.","commonSituations":"Scaling a working single-node IPC setup to multi-node without changing the transport flag; copy-pasted launch scripts across nodes; distributed inference of multimodal models with tensor/data parallelism spanning hosts.","solutions":["Drop --mm-feature-transport cuda_ipc and use the network-capable default transport for multi-node runs","Restrict cuda_ipc usage to single-node deployments (--nnodes 1 or not set)","Add per-topology config: cuda_ipc when nnodes==1, default otherwise"],"exampleFix":"# before\n# node0\npython -m sglang.launch_server --nnodes 2 --node-rank 0 ... --mm-feature-transport cuda_ipc\n# after\npython -m sglang.launch_server --nnodes 2 --node-rank 0 ...  # default transport","handlingStrategy":"validation","validationCode":"def resolve_transport(nnodes: int) -> str | None:\n    if nnodes > 1:\n        return None  # use default network-capable transport\n    return \"cuda_ipc\" if torch.cuda.is_available() else None","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Template launch scripts to inject cuda_ipc only when --nnodes is 1","Keep single-node and multi-node config variants separate instead of one shared flag list"],"tags":["sglang","cuda-ipc","multi-node","multimodal","distributed"],"backgroundTag":"invalid-cli-config-combination","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}