sgl-project/sglang · error · ValueError

attn_sink requires topk_length to be provided as well

Error message

attn_sink requires topk_length to be provided as well

What it means

The optional attention-sink parameter only makes sense with variable-length sparse masks: attn_sink requires topk_length to also be passed. Passing attn_sink without topk_length is rejected because the sink correction is defined relative to per-token masked lengths.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:404

        if topk_length.device != device:
            raise ValueError(
                "topk_length must be on q's device "
                f"{device}, got {topk_length.device}"
            )
        if not topk_length.is_contiguous():
            raise ValueError("topk_length must be contiguous")
        if torch.any(topk_length < 0).item() or torch.any(topk_length > topk).item():
            raise ValueError(
                "topk_length values must satisfy " f"0 <= topk_length <= topk ({topk})"
            )

    if d_v != 512:
        raise ValueError(
            f"sparse_mla_q8kv8_prefill_fwd only supports d_v=512, got {d_v}"
        )

    if attn_sink is not None and topk_length is None:
        raise ValueError("attn_sink requires topk_length to be provided as well")

    if attn_sink is not None:
        if attn_sink.shape != (h_q,) or attn_sink.dtype != torch.float32:
            raise ValueError(
                f"attn_sink must be float32 with shape ({h_q},), got "
                f"{tuple(attn_sink.shape)}/{attn_sink.dtype}"
            )
        if not attn_sink.is_cuda:
            raise ValueError("attn_sink must be a CUDA tensor")
        if attn_sink.device != device:
            raise ValueError(
                f"attn_sink must be on q's device {device}, got {attn_sink.device}"
            )
        if not attn_sink.is_contiguous():
            raise ValueError("attn_sink must be contiguous")

    for name, scale in (("q_scale", q_scale), ("kv_scale", kv_scale)):
        if not isinstance(scale, torch.Tensor):

View on GitHub (pinned to 0132848349)

Solutions

1. Also pass topk_length (int32, shape (s_q,), values in [0, topk]) when enabling attn_sink
2. If you have no variable lengths, pass topk_length full of the padded topk value to emulate fixed-length behavior
3. Or drop attn_sink if sinks are not needed for this call

Example fix

# before
out = fwd(q, kv, indices, attn_sink=sink)
# after
lengths = torch.full((s_q,), indices.shape[-1], dtype=torch.int32, device=q.device)
out = fwd(q, kv, indices, topk_length=lengths, attn_sink=sink)

if attn_sink is not None:
    assert topk_length is not None, "attn_sink requires topk_length"

def sink_args_valid(attn_sink, topk_length) -> bool:
    return attn_sink is None or topk_length is not None

When it happens

Trigger: Calling sparse_mla_q8kv8_prefill_fwd(..., attn_sink=sink) without a topk_length argument (e.g. mirroring a decode-path signature that took attn_sink alone).

Common situations: Adding attention-sink support to a new model integration but reusing the fixed-topk call site; API drift where an older/newer signature allowed attn_sink standalone.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• bad compress_ratio {compress_ratio}
• indices must be on q's device {device}, got {indices.device}
• q must be torch.float8_e4m3fn, got {q.dtype}
• kv must be torch.float8_e4m3fn, got {kv.dtype}
• q must be contiguous

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fd02b156da0a1ab1. Report an issue: GitHub.