{"record":{"id":"f5fde618b2055bfa","repo":"vllm-project/vllm","slug":"padded-n-is-not-supported-with-trtllm-8x4-scale-la","errorCode":null,"errorMessage":"padded_n is not supported with TRTLLM 8x4 scale layout.","messagePattern":"padded_n is not supported with TRTLLM 8x4 scale layout\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/_custom_ops.py","lineNumber":1575,"sourceCode":"    other_dims = 1 if input.ndim == 1 else -1\n    input = input.reshape(other_dims, input.shape[-1])\n    m, n = input.shape\n    block_size = 16\n\n    assert n % block_size == 0, f\"last dim has to be multiple of 16, but got {n}.\"\n    assert input.dtype in (torch.float16, torch.bfloat16), (\n        f\"input.dtype needs to be fp16 or bf16 but got {input.dtype}.\"\n    )\n    if padded_n is not None:\n        assert padded_n >= n, f\"padded_n must be >= n, got padded_n={padded_n}, n={n}.\"\n        assert padded_n % block_size == 0, (\n            f\"padded_n has to be a multiple of {block_size}, but got {padded_n}.\"\n        )\n\n    use_8x4_sf_layout = True if \"trtllm\" in backend and m <= 32 else False  # noqa: SIM210\n    if use_8x4_sf_layout and padded_n is not None and padded_n != n:\n        # TODO: support this case\n        raise ValueError(\"padded_n is not supported with TRTLLM 8x4 scale layout.\")\n    if use_8x4_sf_layout:\n        output, output_scale = flashinfer_quant_nvfp4_8x4_sf_layout(\n            input, input_global_scale\n        )\n    else:\n        # Pre-allocate and call .out variant (same behavior as old in-place API)\n        output, output_scale = create_fp4_output_tensors(\n            m,\n            n,\n            input.device,\n            is_sf_swizzled_layout,\n            padded_n=padded_n,\n        )\n        torch.ops._C.scaled_fp4_quant.out(\n            input,\n            input_global_scale,\n            is_sf_swizzled_layout,\n            output=output,","sourceCodeStart":1557,"sourceCodeEnd":1593,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/_custom_ops.py#L1557-L1593","documentation":"Raised by scaled_fp4_quant() when quantizing NVFP4 activations: the TRTLLM 8x4 scale-factor layout is auto-selected (backend contains 'trtllm' AND m <= 32 rows) while a padded_n different from n was requested. The flashinfer 8x4 kernel used for that layout cannot emit a padded output dimension, so the combination is rejected. It is an explicit TODO: the case is unimplemented, not a hardware error.","triggerScenarios":"Calling vllm._custom_ops.scaled_fp4_quant(input, global_scale, backend='trtllm' or e.g. 'trtllm_gen', padded_n=N) with input reshaped to m<=32 rows (single token / small batch) and padded_n != n.","commonSituations":"Running a TRTLLM-gen quantized model (nvfp4 quant config) during decode with batch size <= 32 where a downstream GEMM wants K padded to a larger alignment; a padded_n larger than n is passed for cublas/trtllm gemm compatibility.","solutions":["Pass padded_n=None (or exactly padded_n == n) when m <= 32 and backend contains 'trtllm'","Use a different backend string (e.g. 'none' or 'cutlass') so the 128x4 swizzled-layout path with padding is taken","Batch to m > 32 so the 8x4 layout is not selected and padded_n works again","If padding is required for the small-batch TRTLLM path, upstream support must be added (see the TODO in _custom_ops.py)"],"exampleFix":"// before\nq, q_scale = ops.scaled_fp4_quant(x, gs, backend=\"trtllm_gen\", padded_n=512)\n// after (small batch decode)\nq, q_scale = ops.scaled_fp4_quant(x, gs, backend=\"trtllm_gen\", padded_n=None)","handlingStrategy":"validation","validationCode":"m, n = input.reshape(-1, input.shape[-1]).shape\nuse_8x4 = \"trtllm\" in backend and m <= 32\nif use_8x4:\n    assert padded_n is None or padded_n == n, \"8x4 SF layout cannot pad; pass padded_n=None\"","typeGuard":null,"tryCatchPattern":"try:\n    q, s = ops.scaled_fp4_quant(x, gs, backend=backend, padded_n=padded_n)\nexcept ValueError as e:\n    if \"8x4\" in str(e):\n        q, s = ops.scaled_fp4_quant(x, gs, backend=backend, padded_n=None)\n    else:\n        raise","preventionTips":["Never combine backend containing 'trtllm' with padded_n for batch<=32 decode paths","Compute m before the call and drop padded_n when m <= 32"],"tags":["quantization","nvfp4","trtllm","gpu","validation"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}