sgl-project/sglang · error · ValueError

--mm-feature-transport=cuda_ipc requires NVIDIA CUDA.

Error message

--mm-feature-transport=cuda_ipc requires NVIDIA CUDA.

What it means

ServerArgs validation rejects --mm-feature-transport=cuda_ipc when the runtime does not detect an NVIDIA CUDA GPU. The cuda_ipc transport passes multimodal feature tensors between processes via CUDA IPC handles, which only exist on NVIDIA GPUs with CUDA drivers. The check runs is_cuda() during server argument resolution before any model loads.

Source

Thrown at python/sglang/srt/server_args.py:9006

                    "--mm-feature-transport=cuda_vmm is not supported with "
                    "SGLANG_RUST_SERVER."
                )
            pool_budget_mb = envs.SGLANG_MM_FEATURE_CACHE_MB.get()
            handle_kind = "CUDA FABRIC" if cfg.nnodes > 1 else "POSIX FD"
            logger.info(
                "Using CUDA VMM for multimodal features with %s sharing: "
                "reserving up to %d MiB on base GPU %d across %d tokenizer "
                "worker(s). This reduces KV cache headroom; a full pool falls "
                "back to inline CPU transport.",
                handle_kind,
                pool_budget_mb,
                cfg.base_gpu_id,
                cfg.tokenizer_worker_num,
            )

        if requested_transport == "cuda_ipc":
            if not is_cuda():
                raise ValueError(
                    "--mm-feature-transport=cuda_ipc requires NVIDIA CUDA."
                )
            if cfg.nnodes != 1:
                raise ValueError(
                    "--mm-feature-transport=cuda_ipc only supports a single node."
                )

            pool_budget_mb = envs.SGLANG_MM_FEATURE_CACHE_MB.get()
            logger.info(
                "Using CUDA IPC for multimodal features: reserving up to %d MiB "
                "on base GPU %d across %d tokenizer worker(s). This reduces KV "
                "cache headroom; a full pool falls back to CPU transport.",
                pool_budget_mb,
                cfg.base_gpu_id,
                cfg.tokenizer_worker_num,
            )
            logger.info(
                "CUDA IPC pool-handle caching is %s. It reuses mappings to the "

View on GitHub (pinned to 0132848349)

Solutions

1. Verify an NVIDIA GPU is visible: run nvidia-smi and check CUDA_VISIBLE_DEVICES isn't empty/masking devices
2. Remove --mm-feature-transport cuda_ipc or switch to the default transport (e.g. --mm-feature-transport cpu_copy or omit the flag)
3. On ROCm/AMD systems, use a supported non-IPC transport for multimodal features

Example fix

# before
python -m sglang.launch_server --model Qwen2.5-VL-7B --mm-feature-transport cuda_ipc
# after (non-CUDA node)
python -m sglang.launch_server --model Qwen2.5-VL-7B

import torch
def can_use_cuda_ipc_transport() -> bool:
    return torch.cuda.is_available() and torch.version.cuda is not None

transport = "cuda_ipc" if can_use_cuda_ipc_transport() else None
# then: launch_server_args.mm_feature_transport = transport

When it happens

Trigger: Launching the SGLang server with --mm-feature-transport cuda_ipc on a machine where is_cuda() returns False (no NVIDIA GPU, ROCm/AMD-only node, CPU-only box, or CUDA driver/device not visible to the process).

Common situations: Running on AMD ROCm or CPU-only dev boxes; CUDA_VISIBLE_DEVICES set to an empty string or hiding all devices; container without GPU passthrough; defaulting the flag in a shared launch script that also runs on non-CUDA nodes.

Related errors

• indices must be on q's device {device}, got {indices.device}
• {selection_error}{component_suffix}
• No compatible attention backend is available{component_suffi
• NCCL only supports CUDA, ROCm and MUSA backends.
• Decode context parallel (decode_context_parallel_size > 1) i

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/08c97a139f5848fb. Report an issue: GitHub.