{"record":{"id":"9256eac54c45a945","repo":"sgl-project/sglang","slug":"gguf-is-selected-by-passing-the-checkpoint-itself-9256ea","errorCode":null,"errorMessage":"GGUF is selected by passing the checkpoint itself, not `--quantization gguf`. Use `--transformer-weights-path <file.gguf>` (or a Hub reference such as owner/repo:Q4_K_M).","messagePattern":"GGUF is selected by passing the checkpoint itself, not `--quantization gguf`\\. Use `--transformer-weights-path <file\\.gguf>` \\(or a Hub reference such as owner/repo:Q4_K_M\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/loader/transformer_load_utils.py","lineNumber":1007,"sourceCode":"    \"\"\"\n    resolve quant config from checkpoints' metadata\n    priority: explicit --quantization flag -> model config.json -> safetensors metadata -> format-specific fallback\n    \"\"\"\n    # priority: explicit --quantization flag (e.g. mxfp8, mxfp4_npu, modelslim)\n    if server_args.quantization is not None:\n        from sglang.multimodal_gen.runtime.layers.quantization import (\n            get_quantization_config,\n        )\n\n        # modelslim requires a per-layer quant description file; load it from\n        # the component directory rather than constructing an empty config.\n        if server_args.quantization == \"modelslim\":\n            return get_quant_config(hf_config, component_model_path)\n\n        # GGUF is selected by pointing at the file, not by this flag: the config\n        # has to be built from that file's header.\n        if server_args.quantization == \"gguf\":\n            raise ValueError(\n                \"GGUF is selected by passing the checkpoint itself, not \"\n                \"`--quantization gguf`. Use \"\n                \"`--transformer-weights-path <file.gguf>` (or a Hub reference \"\n                \"such as owner/repo:Q4_K_M).\"\n            )\n\n        # Online-quant convention: for `fp8`, `mxfp4` and `kitchen_int8`, a\n        # no-arg QuantizationConfig() selects the post-load path -- weights\n        # load in source dtype and are quantized in\n        # process_weights_after_loading.\n        quant_cls = get_quantization_config(server_args.quantization)\n        quant_kwargs = {}\n        if server_args.quantization in {\"fp8\", \"mxfp4\", \"kitchen_int8\"}:\n            quant_kwargs[\"ignored_layers\"] = getattr(\n                server_args, \"quantization_ignored_layers\", None\n            )\n        return quant_cls(**quant_kwargs)\n","sourceCodeStart":989,"sourceCodeEnd":1025,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/loader/transformer_load_utils.py#L989-L1025","documentation":"GGUF checkpoints are not a --quantization mode; their quantization config must be parsed from the GGUF file header itself. The loader therefore rejects the explicit flag and tells you to point at the .gguf file directly.","triggerScenarios":"_resolve_quant_config (via resolve_transformer_quant_load_spec or load_customized) sees server_args.quantization == \"gguf\".","commonSituations":"User familiar with llama.cpp-style tooling assumes --quantization gguf selects the format; or a script enumerates quantization names and passes gguf as one of them.","solutions":["Remove --quantization gguf and pass --transformer-weights-path /path/to/model.gguf","Or use a Hub reference like owner/repo:Q4_K_M as the weights path"],"exampleFix":"# before\n--quantization gguf --model some/model\n# after\n--transformer-weights-path model.gguf --model some/model","handlingStrategy":"validation","validationCode":"def gguf_args_ok(quantization: str | None, weights_path: str | None) -> bool:\n    if weights_path and weights_path.endswith(\".gguf\"):\n        return quantization is None\n    return quantization != \"gguf\"","typeGuard":null,"tryCatchPattern":"try:\n    _resolve_quant_config(...)\nexcept ValueError as e:\n    if \"--quantization gguf\" in str(e):\n        server_args.quantization = None  # route via weights path instead","preventionTips":["Treat GGUF as a file format selected by path, not a quantization enum value","Lint launch scripts that combine --quantization gguf with a non-gguf model path"],"tags":["gguf","quantization","cli-usage","weights-path"],"backgroundTag":"invalid-argument-value","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}