{"record":{"id":"dc7260e012b24066","repo":"sgl-project/sglang","slug":"top-k-must-be-scalar-or-have-one-value-per-row-go","errorCode":null,"errorMessage":"top_k must be scalar or have one value per row, got {top_ks.numel()} values for {batch_size} rows","messagePattern":"top_k must be scalar or have one value per row, got (.+?) values for (.+?) rows","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/sampling/renorm_triton.py","lineNumber":154,"sourceCode":"    probs: torch.Tensor, top_k: Union[torch.Tensor, int]\n) -> torch.Tensor:\n    \"\"\"Apply exact top-k thresholding and renormalize each probability row.\n\n    Sorting uses PyTorch's device kernels because a vocabulary-sized in-register\n    Triton sort does not scale to 100K+ vocabularies. Triton performs the\n    bandwidth-heavy masking, partial reduction, and normalization.\n    \"\"\"\n    probs_fp32 = _prepare_probs(probs)\n    batch_size, vocab_size = probs_fp32.shape\n    if batch_size == 0 or vocab_size == 0:\n        return probs_fp32\n\n    if isinstance(top_k, torch.Tensor):\n        top_ks = top_k.to(device=probs.device, dtype=torch.int64).reshape(-1)\n        if top_ks.numel() == 1:\n            top_ks = top_ks.expand(batch_size)\n        elif top_ks.numel() != batch_size:\n            raise ValueError(\n                f\"top_k must be scalar or have one value per row, got \"\n                f\"{top_ks.numel()} values for {batch_size} rows\"\n            )\n    else:\n        top_ks = torch.full(\n            (batch_size,), int(top_k), device=probs.device, dtype=torch.int64\n        )\n\n    # Match FlashInfer's threshold semantics: sort descending, keep the k highest\n    # probabilities, and retain all ties at the pivot.\n    sorted_probs = torch.sort(probs_fp32, dim=-1, descending=True).values\n    cutoff = (top_ks - 1).clamp_(min=0, max=vocab_size - 1)\n    pivots = sorted_probs.gather(1, cutoff.unsqueeze(1)).squeeze(1).contiguous()\n\n    return _renorm_from_pivots(probs_fp32, pivots)\n\n\n__all__ = [\"top_k_renorm_probs_triton\", \"top_p_renorm_probs_triton\"]","sourceCodeStart":136,"sourceCodeEnd":172,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/sampling/renorm_triton.py#L136-L172","documentation":"top_k_renorm_probs_triton accepts a scalar top_k or a per-row int tensor whose length must equal the batch size. A tensor whose element count is neither 1 nor batch_size cannot be mapped onto rows and is rejected.","triggerScenarios":"Passing a top_k tensor with the wrong length, e.g. [num_requests] while probs has a different batch size, or a 2-D [batch, something>1] tensor.","commonSituations":"Batched sampling where the per-request top_k array was built for a subset of the batch, or leftover top_k tensors from a previous larger batch being reused.","solutions":["Ensure top_k.numel() == 1 or == probs.shape[0]; reshape(-1) first","Rebuild per-row top_k arrays whenever the batch composition changes"],"exampleFix":"# before\ntop_k = torch.full((batch_size + 1,), 50, device='cuda')\nout = top_k_renorm_probs_triton(probs, top_k)\n# after\ntop_k = torch.full((batch_size,), 50, device='cuda')\nout = top_k_renorm_probs_triton(probs, top_k)","handlingStrategy":"validation","validationCode":"if isinstance(top_k, torch.Tensor):\n    assert top_k.numel() in (1, probs.shape[0])","typeGuard":"def top_k_shape_ok(tk, batch): return not isinstance(tk, torch.Tensor) or tk.numel() in (1, batch)","tryCatchPattern":null,"preventionTips":["Rebuild per-request top_k tensors whenever the batch is re-formed"],"tags":["sampling","top-k","batch-size-mismatch","validation"],"backgroundTag":"parameter-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}