What causes "Failed to set up ModelOpt quantization: {e}" in sgl-project/sglang?

Read the chained exception (`raise ... from e`) — the original traceback names the real failure; fix that first

sgl-project/sglang · error · Exception

Failed to set up ModelOpt quantization: {e}

Error message

Failed to set up ModelOpt quantization: {e}

What it means

This is a catch-all wrapper: any exception raised while setting up ModelOpt quantization (config resolution, mtq.quantize, calibration loop, or export) inside the ModelOpt loader is re-raised as Exception('Failed to set up ModelOpt quantization: {e}'). The root cause is the chained original exception `e` — always inspect `__cause__`/the log line above it.

Source

Thrown at python/sglang/srt/model_loader/loader.py:3827

            if not model_parallel_is_initialized() or get_parallel().tp_rank == 0:
                mtq.print_quant_summary(model)

            # Save checkpoint if path provided
            if quantized_ckpt_save_path:
                try:
                    mto.save(model, quantized_ckpt_save_path)
                    rank0_log(f"Quantized model saved to {quantized_ckpt_save_path}")
                except Exception as e:
                    logger.warning(
                        f"Failed to save quantized checkpoint to {quantized_ckpt_save_path}: {e}"
                    )

            # Export model if path provided
            self._maybe_export_modelopt(model, export_path)

        except Exception as e:
            raise Exception(f"Failed to set up ModelOpt quantization: {e}") from e

    def _maybe_export_modelopt(self, model, export_path: str | None) -> None:
        """Export model to HuggingFace format if export_path is provided."""
        if export_path:
            try:
                # Get the original model path from the model config
                original_model_path = getattr(self, "_original_model_path", None)
                self._export_modelopt_checkpoint(
                    model, export_path, original_model_path
                )
                rank0_log(
                    f"Quantized model exported to HuggingFace format at {export_path}"
                )
            except Exception as e:
                rank0_log(
                    f"Warning: Failed to export quantized model to {export_path}: {e}"
                )

View on GitHub (pinned to 0132848349)

Solutions

Read the chained exception (`raise ... from e`) — the original traceback names the real failure; fix that first
Check version compatibility: nvidia-modelopt nightly vs sglang requirements; upgrade/downgrade modelopt
Reduce batch size / calibration samples if the inner error is CUDA OOM
For export errors, verify export_path is writable and on a volume with enough space
If the inner error is unsupported architecture, quantize a supported model or update modelopt

Defensive patterns

Strategy: try-catch

Try / catch

try:
    loader.load_model(...)
except Exception as e:
    root = e.__cause__ or e
    log.error("ModelOpt setup failed, root cause: %s", root)
    raise

Prevention

Always log the chained __cause__, not just the wrapper message
Validate quantization args and model architecture support before starting a long quantization run
Smoke-test modelopt on a tiny model before production quantization

When it happens

Trigger: Calling the ModelOpt loader with an invalid quantization config, a model architecture unsupported by modelopt, calibration data issues, OOM during mtq.quantize, or failures inside _maybe_export_modelopt — any exception in the try block triggers it.

Common situations: Mismatched modelopt vs torch/transformers versions; quantizing a custom model class modelopt doesn't recognize; GPU OOM during FP8/FP4 calibration; export path permission errors; corrupt checkpoint.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/21b9680ffefa5f89. Report an issue: GitHub.