sgl-project/sglang · error · ValueError

Invalid threshold_type for topk: {threshold_type}. Choose 'q

Error message

Invalid threshold_type for topk: {threshold_type}. Choose 'query_head', 'block', or 'overall'.

What it means

moba_attn_varlen validates threshold_type when select_mode == 'topk'. Only 'query_head', 'block', and 'overall' map to a top-k gate-selection routine; any other string reaches the else branch and raises ValueError.

Source

Thrown at python/sglang/multimodal_gen/csrc/attn/vmoba_attn/vmoba/vmoba.py:744

            gate_mask = (valid_gate_mask.flatten() & others_mask_flat).view(gate.shape)
        elif threshold_type == "head_global":
            # per-head top-k across all chunks and sequence positions
            C, H, S = gate.shape
            CS = C * S
            flat_gate = gate.permute(1, 0, 2).reshape(H, CS)
            flat_valid = valid_gate_mask.permute(1, 0, 2).reshape(H, CS)
            flat_gate_masked = torch.where(
                flat_valid, flat_gate, torch.full_like(flat_gate, -float("inf"))
            )
            # pick top-k indices per head
            _, topk_idx = torch.topk(
                flat_gate_masked, k=moba_topk * S, dim=1, largest=True, sorted=False
            )
            gate_idx_flat = torch.zeros_like(flat_valid, dtype=torch.bool)
            gate_idx_flat.scatter_(1, topk_idx, True)
            gate_mask = gate_idx_flat.reshape(H, C, S).permute(1, 0, 2)
        else:
            raise ValueError(
                f"Invalid threshold_type for topk: {threshold_type}. "
                "Choose 'query_head', 'block', or 'overall'."
            )
    elif select_mode == "threshold":
        # Delegate to the specific thresholding function
        valid_gate_mask = gate != -float("inf")  # (num_chunk, num_head, seqlen)
        if threshold_type == "query_head":
            gate_mask = _select_threshold_query_head(
                gate, valid_gate_mask, gate_self_chunk_mask, simsum_threshold
            )
        elif threshold_type == "block":
            gate_mask = _select_threshold_block(
                gate, valid_gate_mask, gate_self_chunk_mask, simsum_threshold
            )
        elif threshold_type == "overall":
            gate_mask = _select_threshold_overall(
                gate, valid_gate_mask, gate_self_chunk_mask, simsum_threshold
            )

View on GitHub (pinned to 0132848349)

Solutions

Set threshold_type to one of 'query_head', 'block', or 'overall' when select_mode='topk'
If you meant per-head global thresholding, switch to select_mode='threshold' with threshold_type='head_global'
Check for typos/whitespace in the config string before calling moba_attn_varlen

Example fix

# before
moba_attn_varlen(q, k, v, ..., select_mode="topk", threshold_type="head_global")
# after
moba_attn_varlen(q, k, v, ..., select_mode="topk", threshold_type="block")

Defensive patterns

Strategy: validation

Validate before calling

VALID_TOPK_THRESHOLD_TYPES = {"query_head", "block", "overall"}
assert select_mode != "topk" or threshold_type in VALID_TOPK_THRESHOLD_TYPES, (
    f"threshold_type must be one of {VALID_TOPK_THRESHOLD_TYPES} for topk mode"
)

Type guard

def is_valid_topk_threshold_type(t: str) -> bool:
    return t in {"query_head", "block", "overall"}

Try / catch

try:
    moba_attn_varlen(..., select_mode="topk", threshold_type=threshold_type)
except ValueError as e:
    if "Invalid threshold_type" in str(e):
        threshold_type = "block"
        moba_attn_varlen(..., select_mode="topk", threshold_type=threshold_type)
    else:
        raise

Prevention

Centralize valid threshold_type strings as module constants
Validate config strings at config-load time, not in the kernel call
Add unit tests covering each valid threshold_type

When it happens

Trigger: Calling moba_attn_varlen(..., select_mode='topk', threshold_type=<anything other than 'query_head'|'block'|'overall'>), e.g. threshold_type='head_global' or a typo like 'queryhead'.

Common situations: Passing a threshold-mode-only value like 'head_global' while select_mode='topk'; renaming/typo in config strings; copy-pasting configs from the threshold code path into topk runs.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7e1d8e9cf8a6a97b. Report an issue: GitHub.