{"record":{"id":"9be0bdd417085ac1","repo":"sgl-project/sglang","slug":"tiny-gemm-no-valid-split-n-for-n-n-k-k-max","errorCode":null,"errorMessage":"tiny_gemm: no valid split_n for N={n}, K={k}, max_m={max_m}; lower max_m","messagePattern":"tiny_gemm: no valid split_n for N=(.+?), K=(.+?), max_m=(.+?); lower max_m","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/gemm/tiny_gemm.py","lineNumber":64,"sourceCode":"\n\ndef _vec_elems() -> int:\n    \"\"\"bf16 elements per vectorized load; mirrors kMaxVecBytes in utils.cuh.\"\"\"\n    from sglang.kernels.jit.utils import get_jit_cuda_arch\n\n    cuda = tuple(int(v) for v in (torch.version.cuda or \"0.0\").split(\".\")[:2])\n    return 16 if get_jit_cuda_arch().major >= 10 and cuda >= (12, 9) else 8\n\n\ndef _default_split_n(n: int, k: int, max_m: int, device: torch.device) -> int:\n    \"\"\"Smallest divisor of n whose n / split_n blocks fit in one wave, subject\n    to the max_m * split_n <= K / vec_elems block-size constraint; falls back\n    to the largest split_n satisfying the constraint (multi-wave grid).\"\"\"\n    sm_count = torch.cuda.get_device_properties(device).multi_processor_count\n    split_cap = (k // _vec_elems()) // max_m\n    divisors = [d for d in range(1, min(n, split_cap) + 1) if n % d == 0]\n    if not divisors:\n        raise RuntimeError(\n            f\"tiny_gemm: no valid split_n for N={n}, K={k}, max_m={max_m};\"\n            \" lower max_m\"\n        )\n    for split in divisors:\n        if n // split <= sm_count:\n            return split\n    return divisors[-1]\n\n\ndef tiny_n_gemm_bf16(\n    x: torch.Tensor,\n    w: torch.Tensor,\n    out: Optional[torch.Tensor] = None,\n    *,\n    out_dtype: Optional[torch.dtype] = None,\n    split_n: Optional[int] = None,\n    max_m: int = _MAX_M_DEFAULT,\n) -> torch.Tensor:","sourceCodeStart":46,"sourceCodeEnd":82,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/gemm/tiny_gemm.py#L46-L82","documentation":"tiny_n_gemm_bf16 splits N so each block respects max_m * split_n <= K / vec_elems; if no divisor of N satisfies the cap (split_cap < 1, i.e. K/vec < max_m), no valid split_n exists. The error suggests lowering max_m.","triggerScenarios":"Calling tiny_n_gemm_bf16 with very small K relative to max_m (e.g. K=256, vec elems 8, max_m=64 gives split_cap=0), so even split_n=1 exceeds the block-size constraint.","commonSituations":"Tiny-K projections (small embedding dims) with default max_m tuned for larger K, or K not a multiple of the vector width wasting budget.","solutions":["Lower max_m (e.g. max_m=16 or 8) so split_cap >= 1.","Ensure K is a multiple of the vector element count for full efficiency.","Fall back to torch.matmul for degenerate tiny shapes."],"exampleFix":"// before\nout = tiny_n_gemm_bf16(x, w, max_m=64)  # K small -> no split\n// after\nout = tiny_n_gemm_bf16(x, w, max_m=8)\n# or\nout = x @ w.t()","handlingStrategy":"fallback","validationCode":"if (k // 8) < max_m:\n    max_m = max(1, k // 8)  # or fall back to torch.matmul","typeGuard":null,"tryCatchPattern":"try:\n    out = tiny_n_gemm_bf16(x, w, max_m=max_m)\nexcept RuntimeError:\n    out = x @ w.t()","preventionTips":["Scale max_m with K: keep max_m <= K/vec_elems.","Fall back to matmul for degenerate tiny-K shapes."],"tags":["gemm","tiny-gemm","split-n","block-size"],"backgroundTag":"invalid-gemm-tile-config","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}