sgl-project/sglang · error · ValueError

Quanto layer {prefix!r} needs a 2D I8 weight, got {data_slic

Error message

Quanto layer {prefix!r} needs a 2D I8 weight, got {data_slice.get_dtype()} {data_shape}

What it means

The packed weight tensor '<prefix>.weight._data' must be a 2D tensor of raw int8 dtype ('I8' in the safetensors file). This error reports the actual dtype and shape when either the dtype is not I8 or the tensor is not 2D.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/configs/quanto_int8_config.py:160

                "input": f"{prefix}.input_scale",
                "output": f"{prefix}.output_scale",
            }
            missing = set(names.values()) - checkpoint_keys
            if missing:
                raise ValueError(
                    f"Quanto layer {prefix!r} is missing tensors: {sorted(missing)}"
                )
            if f"{prefix}.weight" in checkpoint_keys:
                raise ValueError(
                    f"Quanto layer {prefix!r} contains both packed and dense weights"
                )

            data_slice = checkpoint.get_slice(names["data"])
            scale_slice = checkpoint.get_slice(names["scale"])
            data_shape = tuple(data_slice.get_shape())
            scale_shape = tuple(scale_slice.get_shape())
            if data_slice.get_dtype() != "I8" or len(data_shape) != 2:
                raise ValueError(
                    f"Quanto layer {prefix!r} needs a 2D I8 weight, got "
                    f"{data_slice.get_dtype()} {data_shape}"
                )
            if scale_slice.get_dtype() not in _FLOAT_DTYPES or scale_shape != (
                data_shape[0],
                1,
            ):
                raise ValueError(
                    f"Quanto layer {prefix!r} has incompatible scale "
                    f"{scale_slice.get_dtype()} {scale_shape}"
                )
            for scale_name in (names["input"], names["output"]):
                scale = checkpoint.get_slice(scale_name)
                if (
                    scale.get_dtype() not in _FLOAT_DTYPES
                    or tuple(scale.get_shape()) != ()
                ):
                    raise ValueError(

View on GitHub (pinned to 0132848349)

Solutions

1. Confirm data_slice.get_dtype()=='I8' and len(shape)==2 with safetensors before loading
2. Re-quantize so that the target layer is a Linear (2D) layer; exclude conv/other-rank layers from the quantization map
3. If dtype was widened during export, rewrite the tensor as int8 in the safetensors file

Example fix

# validation
sl = ckpt.get_slice(f"{p}.weight._data")
assert sl.get_dtype() == "I8" and len(sl.get_shape()) == 2, (sl.get_dtype(), sl.get_shape())

for p in quantization_map:
    sl = ckpt.get_slice(f'{p}.weight._data')
    if sl.get_dtype() != 'I8' or len(sl.get_shape()) != 2:
        raise SystemExit(f'{p}: bad _data {sl.get_dtype()} {sl.get_shape()}')

def is_valid_i8_2d(sl) -> bool:
    return sl.get_dtype() == 'I8' and len(sl.get_shape()) == 2

try:
    cfg = inspect_quanto_int8_checkpoint(ckpt, mapper)
except ValueError as e:
    if 'needs a 2D I8 weight' in str(e):
        raise SystemExit('Re-export _data as int8 2D; exclude non-linear layers from map')
    raise

When it happens

Trigger: Loading a checkpoint where _data was saved as int32/int16/uint8, or as a higher-rank tensor (e.g. 3D conv weights or a reshaped/export artifact), via inspect_quanto_int8_checkpoint.

Common situations: Quantized 1D-conv/conv3d layers whose packed data isn't 2D, exporting through a format that widened the dtype (e.g. torch.save round-trip or safetensors convert), or hand-crafting _data tensors.

Related errors

• Quanto layer {prefix!r} has incompatible scale {scale_slice.
• {name} must be float32, got {scale.dtype}
• Serialized W4A4 layer {prefix!r} has unsupported linear_dtyp
• QuantoInt8Config must be constructed from safetensors metada
• Quanto checkpoint is missing quantization_map_base64

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d6e1dba00c75a37d. Report an issue: GitHub.