sgl-project/sglang · error · ValueError

Unsupported Quanto weight type for {prefix!r}: {quantization

Error message

Unsupported Quanto weight type for {prefix!r}: {quantization.get('weights')!r}

What it means

inspect_quanto_int8_checkpoint only supports weight-only int8 quantization, so each entry in the checkpoint's quantization_map must specify weights="qint8". Any other weights dtype (e.g. qint4, qint16, float8) raises this error, listing the offending prefix and value.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/configs/quanto_int8_config.py:130

        data_prefixes = {
            name.removesuffix(data_suffix)
            for name in checkpoint_keys
            if name.endswith(data_suffix)
        }
        map_prefixes = set(quantization_map)
        if data_prefixes != map_prefixes:
            missing_map = data_prefixes - map_prefixes
            missing_data = map_prefixes - data_prefixes
            raise ValueError(
                "Quanto tensor/map prefixes do not match: "
                f"missing metadata={sorted(missing_map)[:5]}, "
                f"missing tensors={sorted(missing_data)[:5]}"
            )

        mapped_prefixes: set[str] = set()
        for prefix, quantization in quantization_map.items():
            if quantization.get("weights") != "qint8":
                raise ValueError(
                    f"Unsupported Quanto weight type for {prefix!r}: "
                    f"{quantization.get('weights')!r}"
                )
            if quantization.get("activations") != "none":
                raise ValueError(
                    f"Quanto activation quantization is not supported for {prefix!r}"
                )

            names = {
                "data": f"{prefix}.weight._data",
                "scale": f"{prefix}.weight._scale",
                "input": f"{prefix}.input_scale",
                "output": f"{prefix}.output_scale",
            }
            missing = set(names.values()) - checkpoint_keys
            if missing:
                raise ValueError(
                    f"Quanto layer {prefix!r} is missing tensors: {sorted(missing)}"

View on GitHub (pinned to 0132848349)

Solutions

  1. Re-quantize the model with torchao/quanto qint8 weights (quantize_weights with qint8, no activations)
  2. Use a different quantization config/backend that supports the checkpoint's weight type
  3. Check quantization_map entries: every 'weights' field must read 'qint8' before loading

Example fix

# before
quantize(model, weights=qfloat8)
# after
from torchao.quantization import quant_
quant_(model, Int8WeightOnly())  # produces qint8 weights
Defensive patterns

Strategy: validation

Validate before calling

bad = {p: q for p, q in quantization_map.items() if q.get('weights') != 'qint8'}
if bad: raise SystemExit(f'non-qint8 layers: {list(bad)[:5]}')

Type guard

def is_qint8_map(qmap: dict) -> bool:
    return all(q.get('weights') == 'qint8' for q in qmap.values())

Try / catch

try:
    cfg = inspect_quanto_int8_checkpoint(ckpt, mapper)
except ValueError as e:
    if 'Unsupported Quanto weight type' in str(e):
        raise SystemExit('Re-quantize checkpoint weight-only qint8')
    raise

Prevention

When it happens

Trigger: Loading a checkpoint quantized with quanto to qint4, qint2, float8 or any non-qint8 weight type and passing it to inspect_quanto_int8_checkpoint / an encoder quant config path that calls it.

Common situations: Re-using a checkpoint quantized for a different backend or bit-width, quantizing with quanto.qfloat8/qbits instead of qint8, or swapping in a smaller-precision export to save memory.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/84df311241942731. Report an issue: GitHub.