sgl-project/sglang · error · ValueError

top_p must be scalar or have one value per row, got {top_ps.

Error message

top_p must be scalar or have one value per row, got {top_ps.numel()} values for {batch_size} rows

What it means

top_p_renorm_probs_triton accepts either a scalar top_p or a per-row tensor of thresholds whose length must equal the batch size. A tensor with any other element count (not 1, not batch_size) is ambiguous and rejected.

Source

Thrown at python/sglang/kernels/ops/sampling/renorm_triton.py:111

    probs: torch.Tensor, top_p: Union[torch.Tensor, float]
) -> torch.Tensor:
    """Apply exact top-p thresholding and renormalize each probability row.

    Sorting and prefix sums use PyTorch's device kernels because a vocabulary-sized
    in-register Triton sort does not scale to 100K+ vocabularies. Triton performs
    the bandwidth-heavy masking, partial reduction, and normalization.
    """
    probs_fp32 = _prepare_probs(probs)
    batch_size, vocab_size = probs_fp32.shape
    if batch_size == 0 or vocab_size == 0:
        return probs_fp32

    if isinstance(top_p, torch.Tensor):
        top_ps = top_p.to(device=probs.device, dtype=torch.float32).reshape(-1)
        if top_ps.numel() == 1:
            top_ps = top_ps.expand(batch_size)
        elif top_ps.numel() != batch_size:
            raise ValueError(
                f"top_p must be scalar or have one value per row, got "
                f"{top_ps.numel()} values for {batch_size} rows"
            )
    else:
        if not 0.0 < float(top_p) <= 1.0:
            raise ValueError("top_p values must be in (0, 1]")
        top_ps = torch.full(
            (batch_size,), float(top_p), device=probs.device, dtype=torch.float32
        )

    # Match FlashInfer's threshold semantics: sort ascending, discard the prefix
    # whose cumulative mass is below 1 - p, and retain all ties at the pivot.
    sorted_probs = torch.sort(probs_fp32, dim=-1).values
    cdf = torch.cumsum(sorted_probs, dim=-1)
    cutoff = torch.searchsorted(cdf, (1.0 - top_ps).unsqueeze(1), right=False).squeeze(
        1
    )
    cutoff.clamp_(max=vocab_size - 1)

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure top_p.numel() == 1 or == probs.shape[0]; reshape(-1) shaped [batch] tensors
2. Slice or pad the per-row top_p array to the exact batch size

Example fix

# before
top_p = torch.tensor([0.9, 0.8])  # batch_size = 4
out = top_p_renorm_probs_triton(probs, top_p)
# after
top_p = torch.tensor([0.9, 0.8, 0.9, 0.8])
out = top_p_renorm_probs_triton(probs, top_p)

if isinstance(top_p, torch.Tensor):
    assert top_p.numel() in (1, probs.shape[0])

def top_p_shape_ok(tp, batch): return not isinstance(tp, torch.Tensor) or tp.numel() in (1, batch)

When it happens

Trigger: Passing a top_p tensor of shape [vocab], [batch, 1] left unflattened is fine after reshape, but e.g. [batch//2], [batch, seq], or a list-derived tensor with the wrong length triggers this.

Common situations: Passing per-request top_p arrays misaligned with the prob batch (e.g. batch sliced differently than the top_p array), or forgetting that a broadcast shape [batch,1] flattens to batch (ok) while [1,batch] with batch!=1 mismatches when batch_size!=1.

Related errors

• top_p values must be in (0, 1]
• top_k must be scalar or have one value per row, got {top_ks.
• Input probs contains NaN.
• kv-canary: RealKvSource.read_bytes must be a positive multip
• probs must be 2D, got shape={tuple(probs.shape)}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/766d3b16c80b2c0a. Report an issue: GitHub.