sgl-project/sglang · error · ValueError
GGUF is selected by passing the checkpoint itself, not…
Error message
GGUF is selected by passing the checkpoint itself, not `--quantization gguf`. Use `--transformer-weights-path <file.gguf>` (or a Hub reference such as owner/repo:Q4_K_M).
What it means
GGUF checkpoints are not a --quantization mode; their quantization config must be parsed from the GGUF file header itself. The loader therefore rejects the explicit flag and tells you to point at the .gguf file directly.
Solutions
- Remove --quantization gguf and pass --transformer-weights-path /path/to/model.gguf
- Or use a Hub reference like owner/repo:Q4_K_M as the weights path
Example fix
# before --quantization gguf --model some/model # after --transformer-weights-path model.gguf --model some/model
Defensive patterns
Strategy: validation
Validate before calling
def gguf_args_ok(quantization: str | None, weights_path: str | None) -> bool:
if weights_path and weights_path.endswith(".gguf"):
return quantization is None
return quantization != "gguf" Try / catch
try:
_resolve_quant_config(...)
except ValueError as e:
if "--quantization gguf" in str(e):
server_args.quantization = None # route via weights path instead Prevention
- Treat GGUF as a file format selected by path, not a quantization enum value
- Lint launch scripts that combine --quantization gguf with a non-gguf model path
When it happens
Trigger: _resolve_quant_config (via resolve_transformer_quant_load_spec or load_customized) sees server_args.quantization == "gguf".
Common situations: User familiar with llama.cpp-style tooling assumes --quantization gguf selects the format; or a script enumerates quantization names and passes gguf as one of them.
Understand the failure class
Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.
Related errors
- A GGUF encoder checkpoint cannot be combined with a second…
- Cannot parse checkpoint quantization for
- GGUF and safetensors quantization metadata conflict
- GGUF tensor has inner dimension , which is not a multiple…
- GGUF tensor is not aligned to -element super blocks
AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28).
Data as JSON: /api/errors/9256eac54c45a945.
Report an issue: GitHub.
Appendix: source
Thrown at python/sglang/multimodal_gen/runtime/loader/transformer_load_utils.py:1007
"""
resolve quant config from checkpoints' metadata
priority: explicit --quantization flag -> model config.json -> safetensors metadata -> format-specific fallback
"""
# priority: explicit --quantization flag (e.g. mxfp8, mxfp4_npu, modelslim)
if server_args.quantization is not None:
from sglang.multimodal_gen.runtime.layers.quantization import (
get_quantization_config,
)
# modelslim requires a per-layer quant description file; load it from
# the component directory rather than constructing an empty config.
if server_args.quantization == "modelslim":
return get_quant_config(hf_config, component_model_path)
# GGUF is selected by pointing at the file, not by this flag: the config
# has to be built from that file's header.
if server_args.quantization == "gguf":
raise ValueError(
"GGUF is selected by passing the checkpoint itself, not "
"`--quantization gguf`. Use "
"`--transformer-weights-path <file.gguf>` (or a Hub reference "
"such as owner/repo:Q4_K_M)."
)
# Online-quant convention: for `fp8`, `mxfp4` and `kitchen_int8`, a
# no-arg QuantizationConfig() selects the post-load path -- weights
# load in source dtype and are quantized in
# process_weights_after_loading.
quant_cls = get_quantization_config(server_args.quantization)
quant_kwargs = {}
if server_args.quantization in {"fp8", "mxfp4", "kitchen_int8"}:
quant_kwargs["ignored_layers"] = getattr(
server_args, "quantization_ignored_layers", None
)
return quant_cls(**quant_kwargs)
View on GitHub (pinned to 0132848349)