sgl-project/sglang · error · RuntimeError

attn_res_fused_tma requires SM100+ excluding SM12x; SM{major

Error message

attn_res_fused_tma requires SM100+ excluding SM12x; SM{major}{minor} is unsupported

What it means

attn_res_fused_tma JIT-compiles a warp-specialized TMA kernel that is written for SM100+ (Blackwell) datacenter GPUs, explicitly excluding SM12x (consumer Blackwell). It queries the device capability and raises on any other architecture so an untested GPU never runs a kernel with unsupported TMA/cluster features.

Source

Thrown at python/sglang/kernels/ops/kimi_k3/attn_res.py:40

_DIM: int = 7168  # K3 hidden size, template parameter of the TMA kernel
_MAX_BANK_ROWS: int = 8  # K3 has <= 8 snapshots, upper bound of the nvb dispatch tables


def _make_name(*args):
    return "kimi_k3_attn_res_" + "_".join(str(a) for a in args)


@cache_once
def _jit_fused_tma_module(
    chunk_rows: int, occupancy: int, consumer_regs: int
) -> Module:
    """Compile and cache the warp-specialized TMA aggregation kernel (per-row
    bulk copies into chunk slots; chunk_rows / occupancy / consumer_regs are
    tuning knobs). The smem ring is frozen at 2 chunk slots and PDL is always
    on: the kernel targets SM100+ except SM12x."""
    major, minor = torch.cuda.get_device_capability()
    if major < 10 or major == 12:
        raise RuntimeError(
            "attn_res_fused_tma requires SM100+ excluding SM12x; "
            f"SM{major}{minor} is unsupported"
        )
    args = make_cpp_args(
        _DIM,
        _MAX_BANK_ROWS,
        chunk_rows,
        occupancy,
        consumer_regs,
    )
    with override_jit_cuda_arch(major, minor, suffix="a"):
        return load_jit(
            _make_name("fused_tma"),
            *args,
            cuda_files=["kimi_k3/attn_res/fused_tma.cuh"],
            cuda_wrappers=[
                ("run", f"AttnResFusedTmaKernel<{args}>::run"),
                ("run_pull_rs", f"AttnResFusedTmaKernel<{args}>::run_pull_rs"),

View on GitHub (pinned to 0132848349)

Solutions

Route to a non-TMA fallback (attn_res_fused_direct_ag / pull_rs non-fused path or the plain attention-residual op) when the device is not SM100+
Run on a B200/B100 (SM100) class GPU if the fused TMA path is required
Gate the call behind a check of torch.cuda.get_device_capability() so unsupported GPUs never reach the JIT compile

Example fix

# before
out = attn_res_fused_tma(...)
# after
major, _ = torch.cuda.get_device_capability()
if major >= 10 and major != 12:
    out = attn_res_fused_tma(...)
else:
    out = attn_res_fused_direct_ag(...)  # or non-fused fallback

Defensive patterns

Strategy: fallback

Validate before calling

major, minor = torch.cuda.get_device_capability()
use_fused_tma = major >= 10 and major != 12

Try / catch

try:
    out = attn_res_fused_tma(...)
except RuntimeError as e:
    if 'SM100+' in str(e):
        out = attn_res_fused_direct_ag(...)  # fallback path
    else:
        raise

Prevention

Cache the capability check once at model init and select the backend there
Never assume datacenter Blackwell in multi-GPU heterogenous boxes
Pin deployment hardware in config and assert at startup

When it happens

Trigger: Calling attn_res_fused_tma (or the internal _attn_res_fused_pull_rs_op / _attn_res_fused_direct_ag_op / _precompile) on a GPU with compute capability < 10.0 or in the 12.x family (Hopper SM90, Ada SM89, RTX 50-series SM120, etc.).

Common situations: Running the Kimi K3 attention-residual fused path on H100/A100/RTX 5090 or a CI runner without a Blackwell datacenter GPU; a machine with multiple GPUs where device 0 is not the target architecture.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0a2ba2999cf0e045. Report an issue: GitHub.