{"record":{"id":"bb30cf36b8550b9f","repo":"sgl-project/sglang","slug":"runai-model-streamer-loader-does-not-support-model","errorCode":null,"errorMessage":"Runai Model Streamer Loader does not support ModelOpt quantization yet","messagePattern":"Runai Model Streamer Loader does not support ModelOpt quantization yet","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/model_loader/loader.py","lineNumber":4199,"sourceCode":"            Iterable[RunaiModelStreamerLoader.Source],\n            getattr(model, \"secondary_weights\", ()),\n        )\n        for source in secondary_weights:\n            yield from self._get_weights_iterator(source)\n\n    def download_model(self, model_config: ModelConfig) -> None:\n        self._prepare_weights(model_config.model_path, model_config.revision)\n\n    def load_model(\n        self,\n        *,\n        model_config: ModelConfig,\n        device_config: DeviceConfig,\n    ) -> nn.Module:\n\n        if hasattr(model_config, \"modelopt_quant\") and model_config.modelopt_quant:\n            # Load base model using shared method\n            raise NotImplementedError(\n                \"Runai Model Streamer Loader does not support ModelOpt quantization yet\"\n            )\n\n        assert device_config.device_type in (\"cuda\", \"cpu\"), (\n            f\"Runai Model Streamer only supports CUDA and CPU, \"\n            f\"got {device_config.device_type}\"\n        )\n\n        if device_config.device_type == \"cuda\":\n            self.target_device_str = (\n                device_config.device_type + \":\" + str(device_config.gpu_id)\n            )\n        else:\n            self.target_device_str = \"cpu\"\n\n        target_device = torch.device(device_config.device)\n        quant_config = _get_quantization_config(model_config, self.load_config)\n        with set_default_torch_dtype(model_config.dtype):","sourceCodeStart":4181,"sourceCodeEnd":4217,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/model_loader/loader.py#L4181-L4217","documentation":"RunaiModelStreamerLoader.__init__ or its load path explicitly rejects ModelOpt quantization: if model_config.modelopt_quant is truthy it raises NotImplementedError. The RunAI streaming loader simply hasn't implemented online ModelOpt quantization, unlike the default loader.","triggerScenarios":"Combining --load-format runai_streamer with ModelOpt quantization settings (e.g. --quantization fp8 via modelopt or --json-model-override-args setting modelopt_quant / a modelopt load format flag).","commonSituations":"Trying to speed up loading with the RunAI streamer while also quantizing with ModelOpt; config files reused across setups; MCP/override args setting modelopt_quant implicitly.","solutions":["Drop the ModelOpt quantization flag when using runai_streamer (load a pre-quantized modelopt checkpoint differently)","Or switch to the default loader (--load-format modelopt / default) which supports ModelOpt quantization","Pre-quantize the model offline with modelopt, then load the exported checkpoint with the streamer","Check model_config overrides for a stray modelopt_quant=true"],"exampleFix":"# before\n--load-format runai_streamer --quantization fp8 --modelopt-quant ...\n# NotImplementedError\n\n# after\n--load-format modelopt --quantization fp8  # use the loader that supports modelopt","handlingStrategy":"fallback","validationCode":"if model_config.get(\"modelopt_quant\") and load_format == \"runai_streamer\":\n    raise SystemExit(\"runai_streamer does not support modelopt; use --load-format modelopt\")","typeGuard":"def streamer_supports(model_config: dict, load_format: str) -> bool:\n    return not (model_config.get(\"modelopt_quant\") and load_format == \"runai_streamer\")","tryCatchPattern":"try:\n    loader.load_model(model_config, device_config)\nexcept NotImplementedError:\n    loader = DefaultLoader(...)  # fall back to modelopt-capable loader\n    model = loader.load_model(model_config, device_config)","preventionTips":["Never combine ModelOpt online quantization flags with runai_streamer","Pre-quantize offline, then stream the exported checkpoint","Assert load_format/quantization compatibility in your launch script"],"tags":["runai-streamer","modelopt","not-implemented","unsupported-combination","sglang"],"backgroundTag":"unsupported-feature-combination","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}