sgl-project/sglang · error · ValueError

Quanto layer {prefix!r} has incompatible scale {scale_slice.

Error message

Quanto layer {prefix!r} has incompatible scale {scale_slice.get_dtype()} {scale_shape}

What it means

The per-row scale tensor '<prefix>.weight._scale' must be a float dtype and have shape (out_features, 1) matching the 2D _data's first dimension. This error fires when the scale dtype isn't float or its shape doesn't equal (data_shape[0], 1).

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/configs/quanto_int8_config.py:168

            if f"{prefix}.weight" in checkpoint_keys:
                raise ValueError(
                    f"Quanto layer {prefix!r} contains both packed and dense weights"
                )

            data_slice = checkpoint.get_slice(names["data"])
            scale_slice = checkpoint.get_slice(names["scale"])
            data_shape = tuple(data_slice.get_shape())
            scale_shape = tuple(scale_slice.get_shape())
            if data_slice.get_dtype() != "I8" or len(data_shape) != 2:
                raise ValueError(
                    f"Quanto layer {prefix!r} needs a 2D I8 weight, got "
                    f"{data_slice.get_dtype()} {data_shape}"
                )
            if scale_slice.get_dtype() not in _FLOAT_DTYPES or scale_shape != (
                data_shape[0],
                1,
            ):
                raise ValueError(
                    f"Quanto layer {prefix!r} has incompatible scale "
                    f"{scale_slice.get_dtype()} {scale_shape}"
                )
            for scale_name in (names["input"], names["output"]):
                scale = checkpoint.get_slice(scale_name)
                if (
                    scale.get_dtype() not in _FLOAT_DTYPES
                    or tuple(scale.get_shape()) != ()
                ):
                    raise ValueError(
                        f"Quanto auxiliary scale {scale_name!r} must be a float scalar"
                    )

            mapped_prefix = (
                param_name_mapper(prefix) if param_name_mapper is not None else prefix
            )
            if mapped_prefix in mapped_prefixes:
                raise ValueError(

View on GitHub (pinned to 0132848349)

Solutions

  1. Reshape/save _scale as float32/bfloat16/float16 with shape (out_features, 1)
  2. Ensure the weight was quantized per-row (per output channel) not per-tensor; re-run quanto weight-only int8 quantization if unsure

Example fix

# before: scale saved as scalar (per-tensor)
# 'enc.0.weight._scale': shape ()
# after: per-row scale
scale = scale.reshape(data_shape[0], 1)  # float dtype, e.g. float32
Defensive patterns

Strategy: validation

Validate before calling

for p in quantization_map:
    d = ckpt.get_slice(f'{p}.weight._data'); s = ckpt.get_slice(f'{p}.weight._scale')
    if s.get_dtype() not in ('F32','BF16','F16') or tuple(s.get_shape()) != (d.get_shape()[0], 1):
        raise SystemExit(f'{p}: bad scale {s.get_dtype()} {s.get_shape()}')

Try / catch

try:
    cfg = inspect_quanto_int8_checkpoint(ckpt, mapper)
except ValueError as e:
    if 'incompatible scale' in str(e):
        raise SystemExit('Reshape _scale to (out_features, 1) float')
    raise

Prevention

When it happens

Trigger: A _scale tensor stored as int/half-unsupported dtype, or with shape like (), (1,), (out, out), (1, in) — anything other than (rows, 1) — for a prefix in the quantization_map.

Common situations: Squeezing the scale to a scalar for per-tensor quantization, transposed weight export so the scale axis doesn't match, or a different quanto version writing scales with a different layout.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5c84e15f4be2c1a6. Report an issue: GitHub.