sgl-project/sglang · error · RuntimeError

tiny_gemm: no valid split_n for N={n}, K={k}, max_m={max_m};

Error message

tiny_gemm: no valid split_n for N={n}, K={k}, max_m={max_m}; lower max_m

What it means

tiny_n_gemm_bf16 splits N so each block respects max_m * split_n <= K / vec_elems; if no divisor of N satisfies the cap (split_cap < 1, i.e. K/vec < max_m), no valid split_n exists. The error suggests lowering max_m.

Source

Thrown at python/sglang/kernels/ops/gemm/tiny_gemm.py:64

def _vec_elems() -> int:
    """bf16 elements per vectorized load; mirrors kMaxVecBytes in utils.cuh."""
    from sglang.kernels.jit.utils import get_jit_cuda_arch

    cuda = tuple(int(v) for v in (torch.version.cuda or "0.0").split(".")[:2])
    return 16 if get_jit_cuda_arch().major >= 10 and cuda >= (12, 9) else 8


def _default_split_n(n: int, k: int, max_m: int, device: torch.device) -> int:
    """Smallest divisor of n whose n / split_n blocks fit in one wave, subject
    to the max_m * split_n <= K / vec_elems block-size constraint; falls back
    to the largest split_n satisfying the constraint (multi-wave grid)."""
    sm_count = torch.cuda.get_device_properties(device).multi_processor_count
    split_cap = (k // _vec_elems()) // max_m
    divisors = [d for d in range(1, min(n, split_cap) + 1) if n % d == 0]
    if not divisors:
        raise RuntimeError(
            f"tiny_gemm: no valid split_n for N={n}, K={k}, max_m={max_m};"
            " lower max_m"
        )
    for split in divisors:
        if n // split <= sm_count:
            return split
    return divisors[-1]


def tiny_n_gemm_bf16(
    x: torch.Tensor,
    w: torch.Tensor,
    out: Optional[torch.Tensor] = None,
    *,
    out_dtype: Optional[torch.dtype] = None,
    split_n: Optional[int] = None,
    max_m: int = _MAX_M_DEFAULT,
) -> torch.Tensor:

View on GitHub (pinned to 0132848349)

Solutions

1. Lower max_m (e.g. max_m=16 or 8) so split_cap >= 1.
2. Ensure K is a multiple of the vector element count for full efficiency.
3. Fall back to torch.matmul for degenerate tiny shapes.

Example fix

// before
out = tiny_n_gemm_bf16(x, w, max_m=64)  # K small -> no split
// after
out = tiny_n_gemm_bf16(x, w, max_m=8)
# or
out = x @ w.t()

if (k // 8) < max_m:
    max_m = max(1, k // 8)  # or fall back to torch.matmul

try:
    out = tiny_n_gemm_bf16(x, w, max_m=max_m)
except RuntimeError:
    out = x @ w.t()

When it happens

Trigger: Calling tiny_n_gemm_bf16 with very small K relative to max_m (e.g. K=256, vec elems 8, max_m=64 gives split_cap=0), so even split_n=1 exceeds the block-size constraint.

Common situations: Tiny-K projections (small embedding dims) with default max_m tuned for larger K, or K not a multiple of the vector width wasting budget.

Related errors

• Block sparse tensors{context} require BLOCK_SIZE_KV={base_n_
• Block sparse tensors{context} have block size {sparse_block_
• Block sparse tensors{context} m-block dimension {num_m_block
• cta_n={cta_n} invalid for use_2cta={use_2cta}: bf16 K-major
• TGV cute_ext backend supports {list(_TORCH_TO_CUTLASS_DTYPE)

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9be0bdd417085ac1. Report an issue: GitHub.