{"record":{"id":"9d1a2003e31a8296","repo":"sgl-project/sglang","slug":"rmsnorm-hf-unsupported-hidden-size-hidden-size","errorCode":null,"errorMessage":"rmsnorm_hf: unsupported hidden_size={hidden_size} (must be a multiple of {_WARP_SIZE} in [{_WARP_SIZE}, {_CTA_BLOCK_SIZE}) or a multiple of {_CTA_BLOCK_SIZE})","messagePattern":"rmsnorm_hf: unsupported hidden_size=(.+?) \\(must be a multiple of (.+?) in \\[(.+?), (.+?)\\) or a multiple of (.+?)\\)","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/layernorm/rmsnorm_hf.py","lineNumber":68,"sourceCode":"    input: torch.Tensor,\n    weight: torch.Tensor,\n    eps: float = 1e-6,\n    out: Optional[torch.Tensor] = None,\n) -> torch.Tensor:\n    \"\"\"RMSNorm: ``out = weight * cast_dtype(rsqrt(mean(x^2) + eps) * x)``.\n\n    ``input`` must be 2D ``(num_tokens, hidden_size)``; callers with\n    higher-rank tensors should reshape first. ``hidden_size`` must satisfy\n    :func:`is_supported_rmsnorm_hf_hidden_size`. Empty inputs return an empty\n    output without launching the kernel.\n    \"\"\"\n    if input.dtype not in (torch.float16, torch.bfloat16):\n        raise RuntimeError(f\"rmsnorm_hf: input must be fp16 or bf16, got {input.dtype}\")\n    if input.dim() != 2:\n        raise RuntimeError(f\"rmsnorm_hf: input must be 2D, got {input.dim()}D\")\n    hidden_size = input.size(-1)\n    if not is_supported_rmsnorm_hf_hidden_size(hidden_size):\n        raise RuntimeError(\n            f\"rmsnorm_hf: unsupported hidden_size={hidden_size} \"\n            f\"(must be a multiple of {_WARP_SIZE} in [{_WARP_SIZE}, {_CTA_BLOCK_SIZE}) \"\n            f\"or a multiple of {_CTA_BLOCK_SIZE})\"\n        )\n    if out is None:\n        out = torch.empty_like(input)\n    if input.numel() == 0:\n        return out\n    module = _jit_rmsnorm_hf_module(hidden_size, input.dtype)\n    module.rmsnorm_hf(input, weight, out, eps)\n    return out\n","sourceCodeStart":50,"sourceCodeEnd":80,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/layernorm/rmsnorm_hf.py#L50-L80","documentation":"rmsnorm_hf rejects hidden sizes that are not a multiple of the warp size (32) within [32, CTA_BLOCK_SIZE) or a multiple of the CTA block size. The Triton/CUDA kernel vectorizes loads over the hidden dimension, so arbitrary sizes (e.g. 4095, 1000) cannot be handled. This is a shape-contract error raised before any kernel launch.","triggerScenarios":"Calling rmsnorm_hf(input) with input.size(-1) not a multiple of 32 (when < _CTA_BLOCK_SIZE) or not a multiple of _CTA_BLOCK_SIZE (when larger). Direct calls or via tests with odd hidden sizes.","commonSituations":"Porting a model whose hidden_size is not a power-of-two multiple of 32; slicing the last dim (e.g. x[:, :5120] on a 5122-wide tensor); unit tests iterating arbitrary sizes.","solutions":["Check input.size(-1): round the model hidden size to a multiple of 32 (or the kernel's CTA block size); most real models (4096, 5120, 7168, 8192) already qualify","Pad the last dimension to the next multiple of 32 with torch.nn.functional.pad and slice the output back","If the size is genuinely unsupported, use the generic torch RMSNorm / HF implementation instead of this fused op"],"exampleFix":"// before\nout = rmsnorm_hf(x, weight, eps)  # x: (N, 4095)\n\n// after\npad = (-x.shape[-1]) % 32\nout = rmsnorm_hf(torch.nn.functional.pad(x, (0, pad)), weight, eps)[:, : x.shape[-1]]","handlingStrategy":"validation","validationCode":"from sglang.kernels.ops.layernorm.rmsnorm_hf import is_supported_rmsnorm_hf_hidden_size\nassert is_supported_rmsnorm_hf_hidden_size(x.size(-1)), f\"hidden_size {x.size(-1)} unsupported\"","typeGuard":"def is_rmsnorm_hf_size_ok(x: torch.Tensor) -> bool:\n    return is_supported_rmsnorm_hf_hidden_size(x.size(-1))","tryCatchPattern":null,"preventionTips":["Validate model hidden_size against the kernel's multiple-of-32/CTA-block rule at model load time","Keep unit-test size sweeps to multiples of the block size unless testing rejection"],"tags":["rmsnorm","shape-validation","cuda-kernel","layernorm"],"backgroundTag":"unsupported-tensor-shape","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}