sgl-project/sglang · error · ValueError

Pack: Only supports tensors with dimensions not greater than

Error message

Pack: Only supports tensors with dimensions not greater than 2.

What it means

pack_int4_to_int32 packs an INT4-quantized weight tensor into INT32 words (8 values per word, optionally reordered for the W4A8 layout). The packing logic only handles 1-D or 2-D tensors, so any tensor with ndim > 2 is rejected immediately with ValueError.

Source

Thrown at python/sglang/srt/layers/int4fp8_utils.py:32

    FP8_MAX = 448.0
    scale = w.abs().amax().float() / FP8_MAX
    scaled = (w / scale).clamp(-FP8_MAX, FP8_MAX).to(torch.float8_e4m3fn)
    return scaled, scale


def quantize_int4_scale_columnwise(
    w: torch.Tensor,
) -> Tuple[torch.Tensor, torch.Tensor]:
    S4_MAX = 7
    w_flat = w.reshape(-1, w.shape[-1]).float()
    scale = w_flat.abs().amax(axis=-1) / S4_MAX
    scaled = torch.round(w_flat / scale[:, None]).to(torch.int8).clamp(-S4_MAX, S4_MAX)
    return scaled.reshape(w.shape), scale.reshape(w.shape[:-1])


def pack_int4_to_int32(to_pack: torch.Tensor, reorder: bool = True) -> torch.Tensor:
    if to_pack.ndim > 2:
        raise ValueError(
            "Pack: Only supports tensors with dimensions not greater than 2."
        )

    if reorder:
        order_map = [0, 2, 4, 6, 1, 3, 5, 7]
    else:
        order_map = [0, 1, 2, 3, 4, 5, 6, 7]
    pack_num = 8
    if to_pack.ndim == 2:
        packed = torch.zeros(
            to_pack.shape[0],
            to_pack.shape[1] // pack_num,
            dtype=torch.int32,
            device=to_pack.device,
        )
        new_c = to_pack.shape[1] // pack_num
        for c in range(new_c):
            for i in range(pack_num):

View on GitHub (pinned to 0132848349)

Solutions

  1. Reshape/flatten the tensor to 2-D before packing: to_pack.reshape(-1, to_pack.shape[-1])
  2. If the weight is a fused multi-shard tensor, split it per shard (e.g. separate q/k/v), pack each, then recombine
  3. Check the model's weight_loader passes per-shard 2-D matrices, not stacked 3-D tensors

Example fix

# before
packed = pack_int4_to_int32(w)  # w.shape == (3, 4096, 4096)

# after
packed = torch.stack([pack_int4_to_int32(s) for s in w.unbind(0)])
Defensive patterns

Strategy: validation

Validate before calling

def safe_pack(t: torch.Tensor, reorder: bool = True):
    assert t.ndim <= 2, f"pack_int4_to_int32 expects <=2D, got {tuple(t.shape)}"
    return pack_int4_to_int32(t, reorder=reorder)

Type guard

def is_packable(t: torch.Tensor) -> bool:
    return isinstance(t, torch.Tensor) and t.ndim <= 2

Try / catch

try:
    packed = pack_int4_to_int32(w)
except ValueError:
    packed = torch.stack([pack_int4_to_int32(s) for s in w.reshape(-1, w.shape[-1]).split(w.shape[-2] if w.ndim==3 else 1)])

Prevention

When it happens

Trigger: Calling pack_int4_to_int32 (directly, or via online_int4_fp8_weight_loader on a quantized weight) with a 3-D or higher tensor, e.g. a conv-style weight or an unflattened merged QKV weight of shape [shards, out, in].

Common situations: Loading an INT4-FP8 online-quantized checkpoint whose weight matrices carry an extra leading dimension; model implementations passing fused/stacked weights without first reshaping to 2-D [out_features, in_features].

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5b700439dac0973c. Report an issue: GitHub.