sgl-project/sglang · critical · RuntimeError

Unexpected Ascend TND softmax LSE shape: expected {(q.shape[

Error message

Unexpected Ascend TND softmax LSE shape: expected {(q.shape[0], q.shape[1], 1)}, got {tuple(lse.shape)}

What it means

The Ascend TND attention backend validates the shape of the softmax LSE tensor returned by the NPU flash-attention kernel. After calling the kernel with softmax_lse_flag=True it expects lse to be [num_tokens, num_heads, 1]; any other shape means the kernel contract changed or the packed inputs were malformed.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py:149

        else:
            q, k, v = q.contiguous(), k.contiguous(), v.contiguous()

    output, lse = torch.ops.npu.npu_fused_infer_attention_score(
        q,
        k,
        v,
        num_heads=q.shape[1],
        num_key_value_heads=k.shape[1],
        scale=q.shape[-1] ** -0.5 if softmax_scale is None else softmax_scale,
        input_layout="TND",
        actual_seq_lengths=actual_seq_lengths,
        actual_seq_lengths_kv=actual_seq_lengths_kv,
        softmax_lse_flag=return_softmax_lse,
    )
    if not return_softmax_lse:
        return output
    if lse.shape != (q.shape[0], q.shape[1], 1):
        raise RuntimeError(
            "Unexpected Ascend TND softmax LSE shape: "
            f"expected {(q.shape[0], q.shape[1], 1)}, got {tuple(lse.shape)}"
        )
    return output, lse.squeeze(-1).transpose(0, 1).contiguous()


@dataclass
class AscendFAMetadata:
    pass


class AscendFAMetadataBuilder(AttentionMetadataBuilder):
    def __init__(self) -> None:
        pass

    def prepare(self) -> None:
        pass

View on GitHub (pinned to 0132848349)

Solutions

  1. Print tuple(lse.shape) at the raise site and compare with (q.shape[0], q.shape[1], 1) to identify the actual layout
  2. Pin/roll back the sgkernel-npu (sgl-kernel-npu) version to one this backend was validated against
  3. If it is a permuted/squeezed variant (e.g. [H, T]), add an adapter in the caller that reshapes to (T, H, 1) before consuming it
  4. Report upstream with kernel version and tensor shapes if the kernel contract genuinely changed

Example fix

// before
out, lse = ascend_backend.forward_varlen(q, k, v, cu_seqlens=cu, max_seqlen=m, return_softmax_lse=True)  # RuntimeError
// after: adapt known alternate layout
out, lse = ascend_backend.forward_varlen(q, k, v, cu_seqlens=cu, max_seqlen=m, return_softmax_lse=True)
if lse.dim() == 2:  # e.g. [H, T]
    lse = lse.transpose(0, 1).unsqueeze(-1)  # -> [T, H, 1]
Defensive patterns

Strategy: try-catch

Validate before calling

assert q.ndim == 3, "Ascend TND path requires packed [T, H, D] tensors"
assert q.shape[1] == k.shape[1] or q.shape[1] % k.shape[1] == 0  # sane head config

Type guard

def supports_ascend_lse(backend) -> bool:
    return type(backend).forward_varlen is not AttentionBackend.forward_varlen

Try / catch

try:
    out = backend.forward_varlen(q, k, v, cu_seqlens=cu, max_seqlen=m, return_softmax_lse=True)
except RuntimeError as e:
    if "Unexpected Ascend TND softmax LSE" in str(e):
        out = fallback_backend.forward_varlen(q, k, v, cu_seqlens=cu, max_seqlen=m, return_softmax_lse=True)
    else:
        raise

Prevention

When it happens

Trigger: Calling fused_infer_attention_varlen via forward_varlen or forward_ring_kv_chunk with return_softmax_lse=True on the Ascend backend, where the NPU attentions kernel returns an LSE with unexpected rank or dim ordering.

Common situations: Upgrading/changing sgkernel-npu or the 'attentions' package so the kernel's LSE layout no longer matches; running ring-attention KV merge on Ascend NPU; edge-case head counts or dtypes handled differently by the kernel.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7691f020f94a4880. Report an issue: GitHub.