sgl-project/sglang · error · ValueError

TGV cute_ext backend supports {list(_TORCH_TO_CUTLASS_DTYPE)

Error message

TGV cute_ext backend supports {list(_TORCH_TO_CUTLASS_DTYPE)}; got {dtype}.

What it means

The TGV cute_ext GEMM backend only accepts input dtypes present in _TORCH_TO_CUTLASS_DTYPE (bf16/fp16 mappings). Any other input dtype is rejected before kernel compilation.

Source

Thrown at python/sglang/kernels/ops/gemm/cutedsl_bf16_gemm.py:1140

        dtype,
        c_dtype,
        cta_m,
        cta_n,
        cta_k,
        num_ab_stage,
        bool(use_2cta),
        bool(use_pdl),
        bool(has_bias),
        a_leading,
        b_leading,
        c_leading,
    )
    cached = _TGV_CUTE_EXT_COMPILE_CACHE.get(key)
    if cached is not None:
        return cached

    if dtype not in _TORCH_TO_CUTLASS_DTYPE:
        raise ValueError(
            f"TGV cute_ext backend supports {list(_TORCH_TO_CUTLASS_DTYPE)}; got {dtype}."
        )
    if c_dtype not in _TORCH_TO_CUTLASS_OUT_DTYPE:
        raise ValueError(
            f"TGV cute_ext output supports {list(_TORCH_TO_CUTLASS_OUT_DTYPE)}; "
            f"got {c_dtype}."
        )

    gemm = TgvGemmCuteExtKernel(
        acc_dtype=cutlass.Float32,
        cta_m=cta_m,
        cta_n=cta_n,
        cta_k=cta_k,
        num_ab_stage=num_ab_stage,
        use_2cta=use_2cta,
        use_pdl=use_pdl,
        has_bias=has_bias,
        out_dtype=_TORCH_TO_CUTLASS_OUT_DTYPE[c_dtype],

View on GitHub (pinned to 0132848349)

Solutions

Ensure x and weight are bfloat16 (or the other supported dtype in _TORCH_TO_CUTLASS_DTYPE).
Route fp8 GEMMs to the fp8-specific kernels instead.
Check upstream dtype before dispatch and cast if the math allows.

Example fix

// before
out = _run_tgv(x.float(), w.float(), None)
// after
out = _run_tgv(x.bfloat16(), w.bfloat16(), None)

Defensive patterns

Strategy: type-guard

Validate before calling

assert x.dtype in _TORCH_TO_CUTLASS_DTYPE and weight.dtype in _TORCH_TO_CUTLASS_DTYPE

Type guard

def tgv_input_ok(t: torch.Tensor) -> bool:
    return t.dtype in _TORCH_TO_CUTLASS_DTYPE

Prevention

Route fp8 layers to fp8 kernels, not TGV bf16.
Assert dtypes at dispatch time.

When it happens

Trigger: Calling _run_tgv with x/weight in fp8, fp32, or int dtypes.

Common situations: Routing a quantized (fp8) linear layer to the TGV bf16 path, or accidentally upcasting inputs to fp32.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/cb8173716f9bc695. Report an issue: GitHub.