sgl-project/sglang · error · ValueError

top_p values must be in (0, 1]

Error message

top_p values must be in (0, 1]

What it means

When top_p is given as a Python scalar (not a tensor), the kernel validates the range (0, 1]: p must be positive (p=0 would keep nothing) and at most 1 (p>1 is not a probability mass fraction). Out-of-range scalars are rejected before building the threshold tensor.

Source

Thrown at python/sglang/kernels/ops/sampling/renorm_triton.py:117

    the bandwidth-heavy masking, partial reduction, and normalization.
    """
    probs_fp32 = _prepare_probs(probs)
    batch_size, vocab_size = probs_fp32.shape
    if batch_size == 0 or vocab_size == 0:
        return probs_fp32

    if isinstance(top_p, torch.Tensor):
        top_ps = top_p.to(device=probs.device, dtype=torch.float32).reshape(-1)
        if top_ps.numel() == 1:
            top_ps = top_ps.expand(batch_size)
        elif top_ps.numel() != batch_size:
            raise ValueError(
                f"top_p must be scalar or have one value per row, got "
                f"{top_ps.numel()} values for {batch_size} rows"
            )
    else:
        if not 0.0 < float(top_p) <= 1.0:
            raise ValueError("top_p values must be in (0, 1]")
        top_ps = torch.full(
            (batch_size,), float(top_p), device=probs.device, dtype=torch.float32
        )

    # Match FlashInfer's threshold semantics: sort ascending, discard the prefix
    # whose cumulative mass is below 1 - p, and retain all ties at the pivot.
    sorted_probs = torch.sort(probs_fp32, dim=-1).values
    cdf = torch.cumsum(sorted_probs, dim=-1)
    cutoff = torch.searchsorted(cdf, (1.0 - top_ps).unsqueeze(1), right=False).squeeze(
        1
    )
    cutoff.clamp_(max=vocab_size - 1)
    pivots = sorted_probs.gather(1, cutoff.unsqueeze(1)).squeeze(1).contiguous()

    return _renorm_from_pivots(probs_fp32, pivots)


def top_k_renorm_probs_triton(

View on GitHub (pinned to 0132848349)

Solutions

1. Use a top_p strictly in (0, 1], e.g. 1.0 to disable top-p filtering semantics
2. Treat top_p<=0 in your config as 'no filtering' and substitute 1.0 before calling

Example fix

# before
out = top_p_renorm_probs_triton(probs, 0.0)
# after
top_p = 1.0 if top_p is None or top_p <= 0 else top_p
out = top_p_renorm_probs_triton(probs, top_p)

if not isinstance(top_p, torch.Tensor):
    assert 0.0 < float(top_p) <= 1.0, top_p
# map 'disabled' sentinels to 1.0
top_p = 1.0 if not top_p or top_p <= 0 else top_p

def valid_scalar_top_p(p): return isinstance(p, torch.Tensor) or 0.0 < float(p) <= 1.0

When it happens

Trigger: Calling top_p_renorm_probs_triton with top_p=0.0, a negative value, or something > 1.0; also float('nan') fails the comparison chain.

Common situations: Config typos (top_p=0 meaning 'disabled' in some frameworks but invalid here), misparsed CLI floats, or default-sentinel values like 0 passed through from a sampling-params object.

Related errors

• top_p must be scalar or have one value per row, got {top_ps.
• Input probs contains NaN.
• kv-canary: RealKvSource.read_bytes must be a positive multip
• probs must be 2D, got shape={tuple(probs.shape)}
• renorm kernels require a CUDA/HIP tensor

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a9d216642a078fbd. Report an issue: GitHub.