sgl-project/sglang · error · NotImplementedError

Custom user-provided score_mod is not supported on SM8x arch

Error message

Custom user-provided score_mod is not supported on SM8x architectures.

What it means

Custom user-provided score_mod callables (FlexAttention-style score modification) require kernel features not available on SM8x (Ampere) GPUs; only softcap (converted internally) works there.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attn/cute/interface.py:910

        and int(math.ceil(head_dim / 16) * 16) in [128, 192]
        and int(math.ceil(head_dim_v / 16) * 16) == 128
        and seqlen_q_packgqa > 2 * tile_m
        and (tile_m % qhead_per_kvhead == 0 or not pack_gqa)
        and not qk_blockscaled
    )

    # hd=256 2CTA forward uses dedicated kernel (Blackwell family)
    use_dedicated_hd256_kernel = (
        arch // 10 in [10, 11] and head_dim == 256 and head_dim_v == 256
    )
    use_2cta_instrs = use_2cta_instrs or use_dedicated_hd256_kernel

    if softcap is not None:
        assert score_mod is None, "softcap and score_mod cannot be used together"
        score_mod = utils.create_softcap_scoremod(softcap)
    elif score_mod is not None:
        if arch // 10 == 8:
            raise NotImplementedError(
                "Custom user-provided score_mod is not supported on SM8x architectures."
            )

    # hash score and mask mods for compile cache
    score_mod_hash = utils.hash_callable(score_mod) if score_mod is not None else False
    mask_mod_hash = utils.hash_callable(mask_mod) if mask_mod is not None else False

    is_varlen = (
        cu_seqlens_q is not None
        or cu_seqlens_k is not None
        or seqused_q is not None
        or seqused_k is not None
    )

    # CLC regressed for varlen MHA and dense noncausal. Imbalanced varlen shapes
    # keep more K/V blocks in flight and hurt L2; dense noncausal mostly just
    # pays work-stealing overhead.
    is_varlen_mha = is_varlen and qhead_per_kvhead == 1

View on GitHub (pinned to 0132848349)

Solutions

1. Drop score_mod on SM8x GPUs; express masking via mask_mod/boolean masks if supported
2. Replace score_mod with softcap (still supported, since it's converted internally)
3. Run the workload on SM90+ hardware

Example fix

// before
out = fa(q, k, v, score_mod=my_sigmoid_mod)  # on A100
// after
out = fa(q, k, v)  # or softcap=50.0 instead of custom score_mod

import torch
if score_mod is not None and torch.cuda.get_device_capability()[0] == 8 and not isinstance(score_mod, softcap_mod):
    score_mod = None  # or replace with softcap

def score_mod_supported(arch: int, score_mod) -> bool:
    return score_mod is None or arch // 10 != 8

try:
    out = fa(q, k, v, score_mod=mod)
except NotImplementedError:
    out = fa(q, k, v)  # no score_mod on Ampere

When it happens

Trigger: Calling _flash_attn_fwd/flash_attn_func with a non-None score_mod on a device whose architecture major is 8 (e.g. A100 is fine at 8.0? no — arch//10==8 covers all SM80/86/89).

Common situations: Porting FlexAttention-based masks/score modifications from H100 code to A100/A6000/30xx machines; using capture-based or sigmoid score_mods on Ampere.

Related errors

• Block sparsity + paged KV not supported on SM100
• Block sparsity + sheared bias is not supported on SM90
• Can not import FA3 in sgl_kernel. Please check your installa
• v_cache must be provided
• k_cache can only be None when only_qv=True

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/682f9d881a821efd. Report an issue: GitHub.