{"record":{"id":"5b700439dac0973c","repo":"sgl-project/sglang","slug":"pack-only-supports-tensors-with-dimensions-not-gr","errorCode":null,"errorMessage":"Pack: Only supports tensors with dimensions not greater than 2.","messagePattern":"Pack: Only supports tensors with dimensions not greater than 2\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/int4fp8_utils.py","lineNumber":32,"sourceCode":"    FP8_MAX = 448.0\n    scale = w.abs().amax().float() / FP8_MAX\n    scaled = (w / scale).clamp(-FP8_MAX, FP8_MAX).to(torch.float8_e4m3fn)\n    return scaled, scale\n\n\ndef quantize_int4_scale_columnwise(\n    w: torch.Tensor,\n) -> Tuple[torch.Tensor, torch.Tensor]:\n    S4_MAX = 7\n    w_flat = w.reshape(-1, w.shape[-1]).float()\n    scale = w_flat.abs().amax(axis=-1) / S4_MAX\n    scaled = torch.round(w_flat / scale[:, None]).to(torch.int8).clamp(-S4_MAX, S4_MAX)\n    return scaled.reshape(w.shape), scale.reshape(w.shape[:-1])\n\n\ndef pack_int4_to_int32(to_pack: torch.Tensor, reorder: bool = True) -> torch.Tensor:\n    if to_pack.ndim > 2:\n        raise ValueError(\n            \"Pack: Only supports tensors with dimensions not greater than 2.\"\n        )\n\n    if reorder:\n        order_map = [0, 2, 4, 6, 1, 3, 5, 7]\n    else:\n        order_map = [0, 1, 2, 3, 4, 5, 6, 7]\n    pack_num = 8\n    if to_pack.ndim == 2:\n        packed = torch.zeros(\n            to_pack.shape[0],\n            to_pack.shape[1] // pack_num,\n            dtype=torch.int32,\n            device=to_pack.device,\n        )\n        new_c = to_pack.shape[1] // pack_num\n        for c in range(new_c):\n            for i in range(pack_num):","sourceCodeStart":14,"sourceCodeEnd":50,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/int4fp8_utils.py#L14-L50","documentation":"pack_int4_to_int32 packs an INT4-quantized weight tensor into INT32 words (8 values per word, optionally reordered for the W4A8 layout). The packing logic only handles 1-D or 2-D tensors, so any tensor with ndim > 2 is rejected immediately with ValueError.","triggerScenarios":"Calling pack_int4_to_int32 (directly, or via online_int4_fp8_weight_loader on a quantized weight) with a 3-D or higher tensor, e.g. a conv-style weight or an unflattened merged QKV weight of shape [shards, out, in].","commonSituations":"Loading an INT4-FP8 online-quantized checkpoint whose weight matrices carry an extra leading dimension; model implementations passing fused/stacked weights without first reshaping to 2-D [out_features, in_features].","solutions":["Reshape/flatten the tensor to 2-D before packing: to_pack.reshape(-1, to_pack.shape[-1])","If the weight is a fused multi-shard tensor, split it per shard (e.g. separate q/k/v), pack each, then recombine","Check the model's weight_loader passes per-shard 2-D matrices, not stacked 3-D tensors"],"exampleFix":"# before\npacked = pack_int4_to_int32(w)  # w.shape == (3, 4096, 4096)\n\n# after\npacked = torch.stack([pack_int4_to_int32(s) for s in w.unbind(0)])","handlingStrategy":"validation","validationCode":"def safe_pack(t: torch.Tensor, reorder: bool = True):\n    assert t.ndim <= 2, f\"pack_int4_to_int32 expects <=2D, got {tuple(t.shape)}\"\n    return pack_int4_to_int32(t, reorder=reorder)","typeGuard":"def is_packable(t: torch.Tensor) -> bool:\n    return isinstance(t, torch.Tensor) and t.ndim <= 2","tryCatchPattern":"try:\n    packed = pack_int4_to_int32(w)\nexcept ValueError:\n    packed = torch.stack([pack_int4_to_int32(s) for s in w.reshape(-1, w.shape[-1]).split(w.shape[-2] if w.ndim==3 else 1)])","preventionTips":["Always pass per-shard 2-D [out, in] weight matrices","Reshape fused weights before packing","Add unit tests asserting weight ndim before online quantization"],"tags":["int4","quantization","tensor-shape","packing","weight-loading"],"backgroundTag":"tensor-dimension-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}