sgl-project/sglang · error · NotImplementedError

Runai Model Streamer Loader does not support ModelOpt quanti

Error message

Runai Model Streamer Loader does not support ModelOpt quantization yet

What it means

RunaiModelStreamerLoader.__init__ or its load path explicitly rejects ModelOpt quantization: if model_config.modelopt_quant is truthy it raises NotImplementedError. The RunAI streaming loader simply hasn't implemented online ModelOpt quantization, unlike the default loader.

Source

Thrown at python/sglang/srt/model_loader/loader.py:4199

            Iterable[RunaiModelStreamerLoader.Source],
            getattr(model, "secondary_weights", ()),
        )
        for source in secondary_weights:
            yield from self._get_weights_iterator(source)

    def download_model(self, model_config: ModelConfig) -> None:
        self._prepare_weights(model_config.model_path, model_config.revision)

    def load_model(
        self,
        *,
        model_config: ModelConfig,
        device_config: DeviceConfig,
    ) -> nn.Module:

        if hasattr(model_config, "modelopt_quant") and model_config.modelopt_quant:
            # Load base model using shared method
            raise NotImplementedError(
                "Runai Model Streamer Loader does not support ModelOpt quantization yet"
            )

        assert device_config.device_type in ("cuda", "cpu"), (
            f"Runai Model Streamer only supports CUDA and CPU, "
            f"got {device_config.device_type}"
        )

        if device_config.device_type == "cuda":
            self.target_device_str = (
                device_config.device_type + ":" + str(device_config.gpu_id)
            )
        else:
            self.target_device_str = "cpu"

        target_device = torch.device(device_config.device)
        quant_config = _get_quantization_config(model_config, self.load_config)
        with set_default_torch_dtype(model_config.dtype):

View on GitHub (pinned to 0132848349)

Solutions

1. Drop the ModelOpt quantization flag when using runai_streamer (load a pre-quantized modelopt checkpoint differently)
2. Or switch to the default loader (--load-format modelopt / default) which supports ModelOpt quantization
3. Pre-quantize the model offline with modelopt, then load the exported checkpoint with the streamer
4. Check model_config overrides for a stray modelopt_quant=true

Example fix

# before
--load-format runai_streamer --quantization fp8 --modelopt-quant ...
# NotImplementedError

# after
--load-format modelopt --quantization fp8  # use the loader that supports modelopt

if model_config.get("modelopt_quant") and load_format == "runai_streamer":
    raise SystemExit("runai_streamer does not support modelopt; use --load-format modelopt")

def streamer_supports(model_config: dict, load_format: str) -> bool:
    return not (model_config.get("modelopt_quant") and load_format == "runai_streamer")

try:
    loader.load_model(model_config, device_config)
except NotImplementedError:
    loader = DefaultLoader(...)  # fall back to modelopt-capable loader
    model = loader.load_model(model_config, device_config)

When it happens

Trigger: Combining --load-format runai_streamer with ModelOpt quantization settings (e.g. --quantization fp8 via modelopt or --json-model-override-args setting modelopt_quant / a modelopt load format flag).

Common situations: Trying to speed up loading with the RunAI streamer while also quantizing with ModelOpt; config files reused across setups; MCP/override args setting modelopt_quant implicitly.

Related errors

• quantize_and_serve functionality is currently disabled due t
• quantize_and_serve requires ModelOpt quantization (set with
• Short-conv hybrid models (ZAYA1 CCA, LFM2 / LFM2-MoE) are no
• DSA indexer only supports CUDA, HIP, and NPU
• Mamba2AttnBackend's forward is called directly instead of th

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/bb30cf36b8550b9f. Report an issue: GitHub.