sgl-project/sglang · error · ValueError

SM120 relative bias currently supports head_dim and head_dim

Error message

SM120 relative bias currently supports head_dim and head_dim_v up to 128

What it means

The SM120 FlashAttention relative-position-bias specialization is only implemented for head_dim and head_dim_v up to 128. When has_bias=True and either dim exceeds 128, select_config raises before any kernel is built.

Source

Thrown at python/sglang/kernels/ops/attention/fa4_sm120/runtime.py:627

        tile_mn: Optional[tuple[int, int]],
        has_bias: bool,
        total_q_rows: int,
        num_sms: Optional[int],
        num_batch: int,
        seqlen_q: Optional[int],
        seqlen_k: Optional[int],
        num_head_kv: int,
        qhead_per_kvhead: int,
        is_causal: bool,
        is_local: bool,
        window_size_left: Optional[int],
        window_size_right: Optional[int],
        pack_gqa: bool,
        paged_kv: bool,
    ) -> Sm120ForwardConfig:
        if has_bias:
            if max(head_dim, head_dim_v) > 128:
                raise ValueError(
                    "SM120 relative bias currently supports head_dim and "
                    "head_dim_v up to 128"
                )
            if tile_mn is not None and tile_mn != (64, 128):
                raise ValueError("SM120 relative bias requires tile_mn=(64, 128)")
            tile_m, tile_n = 64, 128
        elif tile_mn is None:
            tile_m, tile_n = FlashAttentionForwardSm120.get_fwd_tile_size(
                head_dim,
                head_dim_v,
                total_q_rows=total_q_rows,
                num_sms=num_sms,
                num_batch=num_batch,
                seqlen_q=seqlen_q,
                seqlen_k=seqlen_k,
                num_head_kv=num_head_kv,
                qhead_per_kvhead=qhead_per_kvhead,
                is_causal=is_causal,

View on GitHub (pinned to 0132848349)

Solutions

Use a backend without the 128-dim bias restriction for head_dim > 128 models
If possible, restructure the model (e.g. split heads) to bring head_dim_v <= 128
Drop the bias or switch to an additive-bias formulation supported at larger dims

Defensive patterns

Strategy: validation

Validate before calling

if has_bias:
    assert max(head_dim, head_dim_v) <= 128, "SM120 bias path supports dims <= 128"

Type guard

def bias_dims_ok(head_dim: int, head_dim_v: int) -> bool:
    return max(head_dim, head_dim_v) <= 128

Prevention

Route bias-enabled large-head-dim models to another backend at init, not at forward time

When it happens

Trigger: Calling _flash_attn_fwd with has_bias=True and max(head_dim, head_dim_v) > 128.

Common situations: Enabling ALiBi/relative bias on a model with head_dim 256 (e.g. some long-context or audio models); porting a bias-enabled workload from another FA backend that allowed it.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7e6d4c2c1df8f45e. Report an issue: GitHub.