sgl-project/sglang · error · ValueError

Quanto auxiliary scale {scale_name!r} must be a float scalar

Error message

Quanto auxiliary scale {scale_name!r} must be a float scalar

What it means

The auxiliary scales '<prefix>.input_scale' and '<prefix>.output_scale' must each be a float-dtype scalar (shape ()). If either has a non-float dtype or any non-scalar shape, this error names the offending tensor.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/configs/quanto_int8_config.py:178

                raise ValueError(
                    f"Quanto layer {prefix!r} needs a 2D I8 weight, got "
                    f"{data_slice.get_dtype()} {data_shape}"
                )
            if scale_slice.get_dtype() not in _FLOAT_DTYPES or scale_shape != (
                data_shape[0],
                1,
            ):
                raise ValueError(
                    f"Quanto layer {prefix!r} has incompatible scale "
                    f"{scale_slice.get_dtype()} {scale_shape}"
                )
            for scale_name in (names["input"], names["output"]):
                scale = checkpoint.get_slice(scale_name)
                if (
                    scale.get_dtype() not in _FLOAT_DTYPES
                    or tuple(scale.get_shape()) != ()
                ):
                    raise ValueError(
                        f"Quanto auxiliary scale {scale_name!r} must be a float scalar"
                    )

            mapped_prefix = (
                param_name_mapper(prefix) if param_name_mapper is not None else prefix
            )
            if mapped_prefix in mapped_prefixes:
                raise ValueError(
                    f"Quanto layers collide after parameter mapping at {mapped_prefix!r}"
                )
            mapped_prefixes.add(mapped_prefix)

    return QuantoInt8Config(mapped_prefixes)


__all__ = ["QuantoInt8Config", "inspect_quanto_int8_checkpoint"]

View on GitHub (pinned to 0132848349)

Solutions

1. Rewrite input_scale/output_scale as 0-d float tensors in the safetensors file (e.g. torch.tensor(v, dtype=torch.float32) with no dims)
2. If an export tool added a leading dim, squeeze it before saving
3. Re-export the checkpoint from the quantized model to regenerate correct scalar scales

Example fix

# before
input_scale = torch.tensor([0.0123])   # shape (1,)
# after
input_scale = torch.tensor(0.0123, dtype=torch.float32)  # shape ()

for p in quantization_map:
    for n in (f'{p}.input_scale', f'{p}.output_scale'):
        sl = ckpt.get_slice(n)
        if sl.get_dtype() not in ('F32','BF16','F16') or tuple(sl.get_shape()) != ():
            raise SystemExit(f'{n}: expected float scalar, got {sl.get_dtype()} {sl.get_shape()}')

def is_float_scalar(sl) -> bool:
    return sl.get_dtype() in ('F32','F16','BF16') and tuple(sl.get_shape()) == ()

try:
    cfg = inspect_quanto_int8_checkpoint(ckpt, mapper)
except ValueError as e:
    if 'must be a float scalar' in str(e):
        raise SystemExit('Squeeze input/output scales to 0-d float tensors')
    raise

When it happens

Trigger: A prefix in the quantization_map whose input_scale/output_scale tensor is stored with shape (1,) or (n,) instead of (), or in an integer dtype.

Common situations: Export pipelines that unsqueeze scalars to (1,) (common when converting through numpy or other frameworks), dtype conversion to float64-int hybrids, or hand-authored scale tensors.

Related errors

• Quanto layer {prefix!r} has incompatible scale {scale_slice.
• QuantoInt8Config must be constructed from safetensors metada
• Quanto checkpoint is missing quantization_map_base64
• Invalid Quanto quantization_map_base64
• Quanto quantization map must be a non-empty object

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0416e3d22055e05b. Report an issue: GitHub.