sgl-project/sglang · error · ValueError

QuantoInt8Config must be constructed from safetensors metada

Error message

QuantoInt8Config must be constructed from safetensors metadata

What it means

QuantoInt8Config.from_config is intentionally disabled (raises always) and get_config_filenames returns []. Quanto int8 metadata lives in safetensors headers, not a JSON config file, so the config must be built via inspect_quanto_int8_checkpoint instead of the standard from_config path.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/configs/quanto_int8_config.py:63

    @classmethod
    def get_name(cls) -> str:
        return "quanto_int8"

    @classmethod
    def get_supported_act_dtypes(cls) -> list[torch.dtype]:
        return [torch.bfloat16, torch.float16]

    @classmethod
    def get_min_capability(cls) -> int:
        return 0

    @staticmethod
    def get_config_filenames() -> list[str]:
        return []

    @classmethod
    def from_config(cls, config: dict[str, Any]) -> QuantoInt8Config:
        raise ValueError(
            "QuantoInt8Config must be constructed from safetensors metadata"
        )

    def get_quant_method(
        self, layer: torch.nn.Module, prefix: str
    ) -> QuantizeMethodBase | None:
        if isinstance(layer, DiffusionLinearBase):
            unquantized_method = DiffusionUnquantizedLinearMethod
        elif isinstance(layer, SrtLinearBase):
            unquantized_method = SrtUnquantizedLinearMethod
        else:
            return None
        if prefix not in self.layer_prefixes:
            return unquantized_method()
        self.selected.add(prefix)
        return QuantoInt8LinearMethod()

View on GitHub (pinned to 0132848349)

Solutions

1. Call inspect_quanto_int8_checkpoint(safetensors_file) to build the config from checkpoint metadata
2. Special-case quanto checkpoints in your loader before falling back to from_config
3. Check checkpoint metadata['quantization_format'] == 'quanto' first

Example fix

# before
cfg = QuantoInt8Config.from_config({})  # always raises

# after
from safetensors import safe_open
with safe_open(path, framework="pt") as f:
    cfg = inspect_quanto_int8_checkpoint(f)

from safetensors import safe_open
with safe_open(ckpt_path, framework="pt") as f:
    if (f.metadata() or {}).get("quantization_format") == "quanto":
        cfg = inspect_quanto_int8_checkpoint(f)
    else:
        cfg = SomeOtherConfig.from_config(config_dict)

try:
    cfg = QuantoInt8Config.from_config(d)
except ValueError:
    cfg = inspect_quanto_int8_checkpoint(open_safetensors(path))

When it happens

Trigger: Any code path (e.g. a generic loader calling QuantizationConfig.from_config(config_dict)) that tries to deserialize QuantoInt8Config from a hf quantization_config dict.

Common situations: Plugging QuantoInt8Config into a framework that auto-instantiates configs via from_config; forgetting to special-case quanto detection before the generic loader.

Related errors

• Quanto checkpoint is missing quantization_map_base64
• Invalid Quanto quantization_map_base64
• Quanto quantization map must be a non-empty object
• Quanto quantization map entries must be named objects
• Quanto tensor/map prefixes do not match: missing metadata={s

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c3eea3244be8e114. Report an issue: GitHub.