sgl-project/sglang · error · RuntimeError

Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `

Error message

Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `nvidia-mathdx` package (`pip install nvidia-mathdx`) or set MATHDX_HOME to an extracted Math-DX archive root.

What it means

The SGLang JIT kernel build system cannot locate NVIDIA Math-DX (cuBLASDx) headers, which are required to compile JIT kernels that use cuBLASDx. The resolver first checks the MATHDX_HOME environment variable, then looks for an installed nvidia-mathdx pip package or extracted archive; when none is found it raises this RuntimeError from get_mathdx_include_paths().

Source

Thrown at python/sglang/kernels/jit/utils/deps.py:94

    # handles regular packages.
    spec = importlib.util.find_spec("nvidia.mathdx")
    if spec is not None:
        roots = list(spec.submodule_search_locations or [])
        if spec.origin is not None:
            roots.append(str(pathlib.Path(spec.origin).parent))
        for root in roots:
            candidate = pathlib.Path(root).resolve()
            if (candidate / "include").exists():
                return candidate

    return None


@register_dependency("mathdx")
def get_mathdx_include_paths() -> List[str]:
    root = get_mathdx_root()
    if root is None:
        raise RuntimeError(
            "Cannot find NVIDIA Math-DX (cuBLASDx) headers. "
            "Install the `nvidia-mathdx` package "
            "(`pip install nvidia-mathdx`) or set MATHDX_HOME to an "
            "extracted Math-DX archive root."
        )
    candidates = [root / "include"]
    cutlass = root / "external" / "cutlass" / "include"
    if cutlass.exists():
        candidates.append(cutlass)
    return [str(p) for p in candidates]


@register_dependency("cutlass")
def get_cutlass_include_paths() -> List[str]:
    include_paths: List[str] = []

    flashinfer_root = _find_package_root("flashinfer")
    if flashinfer_root is not None:

View on GitHub (pinned to 0132848349)

Solutions

pip install nvidia-mathdx (matches your installed CUDA major version)
Set MATHDX_HOME to the root of an extracted Math-DX archive (a directory containing include/), e.g. export MATHDX_HOME=/opt/mathdx
If behind a firewall, download the Math-DX archive from NVIDIA, extract it, and point MATHDX_HOME at it
Verify by running python -c 'from sglang.kernels.jit.utils.deps import get_mathdx_root; print(get_mathdx_root())' and ensure it returns a path containing include/cublasdx

Example fix

# before: MATHDX_HOME unset, nvidia-mathdx not installed -> RuntimeError
pip install nvidia-mathdx
# after
export MATHDX_HOME=/opt/nvidia/mathdx  # or rely on the pip package
python -c "from sglang.kernels.jit.utils.deps import get_mathdx_include_paths; print(get_mathdx_include_paths())"

Defensive patterns

Strategy: validation

Validate before calling

import os
from sglang.kernels.jit.utils.deps import get_mathdx_root
if get_mathdx_root() is None and not os.environ.get('MATHDX_HOME'):
    raise SystemExit('Install nvidia-mathdx or set MATHDX_HOME before building JIT kernels')

Prevention

Pin nvidia-mathdx in the environment alongside the matching CUDA version
Set MATHDX_HOME in Dockerfile/CI env so kernel builds never depend on wheel layout
Add an env-check smoke test that calls get_mathdx_root() before the first JIT build

When it happens

Trigger: Building/compiling a JIT kernel registered with the @register_dependency("mathdx") decorator (any kernel whose CMake/source list calls get_mathdx_include_paths()) on a machine without the nvidia-mathdx wheel installed and without MATHDX_HOME pointing at an extracted Math-DX archive root.

Common situations: Fresh CI runner or Docker image that installs sglang but not the optional nvidia-mathdx dependency; GPU driver/CUDA stack upgraded and a previously working MATHDX_HOME path no longer exists; using a CUDA version for which the nvidia-mathdx wheel was not installed via pip index.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6bb0911fbc46acca. Report an issue: GitHub.