sgl-project/sglang · error · ValueError

Found unknown quantization='{quantization}' in config

Error message

Found unknown quantization='{quantization}' in config

What it means

While adapting a Mistral-format params.json config to HF style, the quantization field value does not match any known scheme (fp8 dynamic/static, etc.), so it cannot be remapped into quantization_config.

Source

Thrown at python/sglang/srt/utils/hf_transformers/mistral_utils.py:237

def _remap_mistral_quantization_args(config: dict) -> dict:
    if config.get("quantization"):
        quantization = config.pop("quantization", {})
        if quantization.get("qformat_weight") == "fp8_e4m3":
            qscheme_act = quantization.get("qscheme_act")
            assert qscheme_act in (
                "NO_SCALES",
                "TENSOR",
                None,
            ), "Only NO_SCALES and TENSOR (default) are supported for qscheme_act"
            is_dynamic = qscheme_act == "NO_SCALES"
            config["quantization_config"] = {
                "quant_method": "fp8",
                "activation_scheme": "dynamic" if is_dynamic else "static",
            }
        else:
            raise ValueError(f"Found unknown quantization='{quantization}' in config")

    return config


def _remap_mistral_audio_args(config: dict) -> dict:
    whisper_args = config["multimodal"].pop("whisper_model_args")
    encoder_args = whisper_args["encoder_args"]
    downsample_args = whisper_args["downsample_args"]

    quant_config = config.get("quantization_config")
    config = {
        "model_type": "whixtral",
        "architectures": ["VoxtralForConditionalGeneration"],
        "text_config": PretrainedConfig.from_dict(config),
        "audio_config": WhisperConfig(
            num_mel_bins=encoder_args["audio_encoding_args"]["num_mel_bins"],
            window_size=encoder_args["audio_encoding_args"]["window_size"],
            sampling_rate=encoder_args["audio_encoding_args"]["sampling_rate"],

View on GitHub (pinned to 0132848349)

Solutions

1. Upgrade sglang to a version supporting the quant scheme
2. Serve the HF-format version of the model instead of Mistral format
3. Remove/bypass the quantization entry only if the weights are actually unquantized

q = params.get('quantization'); known = {None,'fp8','fp8_dynamic', ...}
if q not in known: prefer HF-format model

try:
    adapt_config_dict(cfg)
except ValueError as e:
    if 'unknown quantization' in str(e): use_hf_checkpoint()

When it happens

Trigger: Loading a Mistral-format model whose params.json declares a new or unrecognized quantization string (e.g. a newer int4/fp4 variant).

Common situations: Mistral releases a new quant format before sglang adds a remapping; custom-quantized Mistral models.

Related errors

• The quantization method `{quantization}` is already exists.
• The quantization config must be a subclass of `QuantizationC
• Invalid quantization method: {quantization}
• kitchen_w4a4 is inferred from per-layer checkpoint metadata;
• GGUFConfig must be constructed from a GGUF checkpoint

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/eb84c7880ce85588. Report an issue: GitHub.