{"record":{"id":"84df311241942731","repo":"sgl-project/sglang","slug":"unsupported-quanto-weight-type-for-prefix-r-qu","errorCode":null,"errorMessage":"Unsupported Quanto weight type for {prefix!r}: {quantization.get('weights')!r}","messagePattern":"Unsupported Quanto weight type for (.+?): (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/quantization/configs/quanto_int8_config.py","lineNumber":130,"sourceCode":"        data_prefixes = {\n            name.removesuffix(data_suffix)\n            for name in checkpoint_keys\n            if name.endswith(data_suffix)\n        }\n        map_prefixes = set(quantization_map)\n        if data_prefixes != map_prefixes:\n            missing_map = data_prefixes - map_prefixes\n            missing_data = map_prefixes - data_prefixes\n            raise ValueError(\n                \"Quanto tensor/map prefixes do not match: \"\n                f\"missing metadata={sorted(missing_map)[:5]}, \"\n                f\"missing tensors={sorted(missing_data)[:5]}\"\n            )\n\n        mapped_prefixes: set[str] = set()\n        for prefix, quantization in quantization_map.items():\n            if quantization.get(\"weights\") != \"qint8\":\n                raise ValueError(\n                    f\"Unsupported Quanto weight type for {prefix!r}: \"\n                    f\"{quantization.get('weights')!r}\"\n                )\n            if quantization.get(\"activations\") != \"none\":\n                raise ValueError(\n                    f\"Quanto activation quantization is not supported for {prefix!r}\"\n                )\n\n            names = {\n                \"data\": f\"{prefix}.weight._data\",\n                \"scale\": f\"{prefix}.weight._scale\",\n                \"input\": f\"{prefix}.input_scale\",\n                \"output\": f\"{prefix}.output_scale\",\n            }\n            missing = set(names.values()) - checkpoint_keys\n            if missing:\n                raise ValueError(\n                    f\"Quanto layer {prefix!r} is missing tensors: {sorted(missing)}\"","sourceCodeStart":112,"sourceCodeEnd":148,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/quantization/configs/quanto_int8_config.py#L112-L148","documentation":"inspect_quanto_int8_checkpoint only supports weight-only int8 quantization, so each entry in the checkpoint's quantization_map must specify weights=\"qint8\". Any other weights dtype (e.g. qint4, qint16, float8) raises this error, listing the offending prefix and value.","triggerScenarios":"Loading a checkpoint quantized with quanto to qint4, qint2, float8 or any non-qint8 weight type and passing it to inspect_quanto_int8_checkpoint / an encoder quant config path that calls it.","commonSituations":"Re-using a checkpoint quantized for a different backend or bit-width, quantizing with quanto.qfloat8/qbits instead of qint8, or swapping in a smaller-precision export to save memory.","solutions":["Re-quantize the model with torchao/quanto qint8 weights (quantize_weights with qint8, no activations)","Use a different quantization config/backend that supports the checkpoint's weight type","Check quantization_map entries: every 'weights' field must read 'qint8' before loading"],"exampleFix":"# before\nquantize(model, weights=qfloat8)\n# after\nfrom torchao.quantization import quant_\nquant_(model, Int8WeightOnly())  # produces qint8 weights","handlingStrategy":"validation","validationCode":"bad = {p: q for p, q in quantization_map.items() if q.get('weights') != 'qint8'}\nif bad: raise SystemExit(f'non-qint8 layers: {list(bad)[:5]}')","typeGuard":"def is_qint8_map(qmap: dict) -> bool:\n    return all(q.get('weights') == 'qint8' for q in qmap.values())","tryCatchPattern":"try:\n    cfg = inspect_quanto_int8_checkpoint(ckpt, mapper)\nexcept ValueError as e:\n    if 'Unsupported Quanto weight type' in str(e):\n        raise SystemExit('Re-quantize checkpoint weight-only qint8')\n    raise","preventionTips":["Standardize on weight-only qint8 when producing checkpoints for this runtime","Document the supported quant recipe (weights=qint8, activations=none) in model export scripts"],"tags":["quantization","quanto","int8","unsupported-dtype"],"backgroundTag":"unsupported-quantization-scheme","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}