{"record":{"id":"766d3b16c80b2c0a","repo":"sgl-project/sglang","slug":"top-p-must-be-scalar-or-have-one-value-per-row-go","errorCode":null,"errorMessage":"top_p must be scalar or have one value per row, got {top_ps.numel()} values for {batch_size} rows","messagePattern":"top_p must be scalar or have one value per row, got (.+?) values for (.+?) rows","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/sampling/renorm_triton.py","lineNumber":111,"sourceCode":"    probs: torch.Tensor, top_p: Union[torch.Tensor, float]\n) -> torch.Tensor:\n    \"\"\"Apply exact top-p thresholding and renormalize each probability row.\n\n    Sorting and prefix sums use PyTorch's device kernels because a vocabulary-sized\n    in-register Triton sort does not scale to 100K+ vocabularies. Triton performs\n    the bandwidth-heavy masking, partial reduction, and normalization.\n    \"\"\"\n    probs_fp32 = _prepare_probs(probs)\n    batch_size, vocab_size = probs_fp32.shape\n    if batch_size == 0 or vocab_size == 0:\n        return probs_fp32\n\n    if isinstance(top_p, torch.Tensor):\n        top_ps = top_p.to(device=probs.device, dtype=torch.float32).reshape(-1)\n        if top_ps.numel() == 1:\n            top_ps = top_ps.expand(batch_size)\n        elif top_ps.numel() != batch_size:\n            raise ValueError(\n                f\"top_p must be scalar or have one value per row, got \"\n                f\"{top_ps.numel()} values for {batch_size} rows\"\n            )\n    else:\n        if not 0.0 < float(top_p) <= 1.0:\n            raise ValueError(\"top_p values must be in (0, 1]\")\n        top_ps = torch.full(\n            (batch_size,), float(top_p), device=probs.device, dtype=torch.float32\n        )\n\n    # Match FlashInfer's threshold semantics: sort ascending, discard the prefix\n    # whose cumulative mass is below 1 - p, and retain all ties at the pivot.\n    sorted_probs = torch.sort(probs_fp32, dim=-1).values\n    cdf = torch.cumsum(sorted_probs, dim=-1)\n    cutoff = torch.searchsorted(cdf, (1.0 - top_ps).unsqueeze(1), right=False).squeeze(\n        1\n    )\n    cutoff.clamp_(max=vocab_size - 1)","sourceCodeStart":93,"sourceCodeEnd":129,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/sampling/renorm_triton.py#L93-L129","documentation":"top_p_renorm_probs_triton accepts either a scalar top_p or a per-row tensor of thresholds whose length must equal the batch size. A tensor with any other element count (not 1, not batch_size) is ambiguous and rejected.","triggerScenarios":"Passing a top_p tensor of shape [vocab], [batch, 1] left unflattened is fine after reshape, but e.g. [batch//2], [batch, seq], or a list-derived tensor with the wrong length triggers this.","commonSituations":"Passing per-request top_p arrays misaligned with the prob batch (e.g. batch sliced differently than the top_p array), or forgetting that a broadcast shape [batch,1] flattens to batch (ok) while [1,batch] with batch!=1 mismatches when batch_size!=1.","solutions":["Ensure top_p.numel() == 1 or == probs.shape[0]; reshape(-1) shaped [batch] tensors","Slice or pad the per-row top_p array to the exact batch size"],"exampleFix":"# before\ntop_p = torch.tensor([0.9, 0.8])  # batch_size = 4\nout = top_p_renorm_probs_triton(probs, top_p)\n# after\ntop_p = torch.tensor([0.9, 0.8, 0.9, 0.8])\nout = top_p_renorm_probs_triton(probs, top_p)","handlingStrategy":"validation","validationCode":"if isinstance(top_p, torch.Tensor):\n    assert top_p.numel() in (1, probs.shape[0])","typeGuard":"def top_p_shape_ok(tp, batch): return not isinstance(tp, torch.Tensor) or tp.numel() in (1, batch)","tryCatchPattern":null,"preventionTips":["Build per-row top_p arrays from the same request list as the batch"],"tags":["sampling","top-p","batch-size-mismatch","validation"],"backgroundTag":"parameter-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}