vllm-project/vllm · error · RuntimeError

The `_qutlass_C` extension is not loaded. Make sure your cus

Error message

The `_qutlass_C` extension is not loaded. Make sure your custom op library is imported before calling fusedQuantizeMx.

What it means

fusedQuantizeMx() dispatches to torch.ops._qutlass_C.fusedQuantizeMxQuest / fusedQuantizeMxAbsMax. That namespace is only registered when the CUTLASS-based _qutlass_C extension shared library has been imported/loaded; if it is not, the wrapper raises RuntimeError instead of letting torch.ops raise an opaque AttributeError.

Source

Thrown at vllm/_custom_ops.py:4117

    if b.device != a.device:
        raise ValueError("`a` and `b` must be on the same device.")

    xh_e2m1 = torch.empty(
        *a.shape[:-1], a.size(-1) // 2, dtype=torch.uint8, device=a.device
    )

    rows, cols = a.numel() // a.size(-1), a.size(-1) // 32
    n_row_blocks = cdiv(rows, 128)
    n_col_blocks = cdiv(cols, 4)
    padded_rows = n_row_blocks * 128
    padded_cols = n_col_blocks * 4

    xh_e8m0 = torch.empty(
        padded_rows, padded_cols, dtype=torch.float8_e8m0fnu, device=a.device
    )

    if not hasattr(torch.ops, "_qutlass_C"):
        raise RuntimeError(
            "The `_qutlass_C` extension is not loaded. "
            "Make sure your custom op library is imported before calling fusedQuantizeMx."
        )

    if method == "quest":
        return torch.ops._qutlass_C.fusedQuantizeMxQuest(a, b, xh_e2m1, xh_e8m0)
    elif method == "abs_max":
        return torch.ops._qutlass_C.fusedQuantizeMxAbsMax(a, b, xh_e2m1, xh_e8m0)
    else:
        raise ValueError(f"invalid method {method!r}, must be 'quest' or 'abs_max'")


if hasattr(torch.ops._qutlass_C, "fusedQuantizeNvAbsMax"):

    @register_fake("_qutlass_C::fusedQuantizeNvAbsMax")
    def _fake_fused_quantize_nv_absmax(
        a: torch.Tensor,
        b: torch.Tensor,

View on GitHub (pinned to c794754062)

Solutions

import vllm (or the module that loads custom op libraries, e.g. vllm._quatlss_C loader) before calling the op
Verify the extension loads: check hasattr(torch.ops, '_qutlass_C') after importing vllm; reinstall vLLM if absent on a CUDA build
Skip the code path on platforms where the extension is not built

Example fix

# before
from vllm import _custom_ops as ops
q, s = ops.fusedQuantizeMx(a, b)  # _qutlass_C never loaded
# after
import vllm  # loads custom op libraries first
from vllm import _custom_ops as ops
assert hasattr(torch.ops, "_qutlass_C")
q, s = ops.fusedQuantizeMx(a, b)

Defensive patterns

Strategy: type-guard

Validate before calling

import torch, vllm
if not hasattr(torch.ops, "_qutlass_C"):
    raise RuntimeError("build lacks _qutlass_C; use a CUDA vLLM build")

Type guard

def fused_quantize_mx_available() -> bool:
    import torch, vllm  # vllm import loads op libraries
    return hasattr(torch.ops, "_qutlass_C")

Try / catch

try:
    q, s = ops.fusedQuantizeMx(a, b, method="abs_max")
except RuntimeError as e:
    if "_qutlass_C" in str(e):
        q, s = torch_ref_mx_quant(a)  # reference path
    else:
        raise

Prevention

Import vllm before touching _custom_ops in scripts
Gate exotic quant paths on hasattr(torch.ops, '_qutlass_C')

When it happens

Trigger: Calling vllm._custom_ops.fusedQuantizeMx() in an environment where the _qutlass_C extension was never loaded — CPU-only vLLM wheel, a stripped build, or calling before the module that registers the ops (vllm import chain) has run.

Common situations: Unit-testing the wrapper on a machine without the compiled extension; importing _custom_ops directly in a minimal script without importing vllm first; a broken/partial wheel install where the .so is missing.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/7a123d15c1dd715e. Report an issue: GitHub.