sgl-project/sglang · error · NotImplementedError

GGUF models are not supported.

Error message

GGUF models are not supported.

What it means

get_hf_config refuses to load configs from GGUF-quantized checkpoints; only standard HF/diffusers formats are supported. It detects a GGUF file via check_gguf_file on the model path and raises NotImplementedError immediately.

Source

Thrown at python/sglang/multimodal_gen/runtime/utils/hf_diffusers_utils.py:431

        AutoConfig.register(name, cls)


def download_from_hf(model_path: str):
    if os.path.exists(model_path):
        return model_path

    return snapshot_download(model_path, allow_patterns=["*.json", "*.bin", "*.model"])


def get_hf_config(
    component_model_path: str,
    trust_remote_code: bool,
    revision: str | None = None,
    model_override_args: dict | None = None,
    **kwargs,
) -> PretrainedConfig:
    if check_gguf_file(component_model_path):
        raise NotImplementedError("GGUF models are not supported.")

    config = AutoConfig.from_pretrained(
        component_model_path,
        trust_remote_code=trust_remote_code,
        revision=revision,
        **kwargs,
    )
    if config.model_type in _CONFIG_REGISTRY:
        config_class = _CONFIG_REGISTRY[config.model_type]
        config = config_class.from_pretrained(component_model_path, revision=revision)
        # NOTE(HandH1998): Qwen2VL requires `_name_or_path` attribute in `config`.
        config._name_or_path = component_model_path
    if model_override_args:
        config.update(model_override_args)

    return config

View on GitHub (pinned to 0132848349)

Solutions

1. Use the original HF or diffusers-format model instead of the GGUF quantization
2. If quantization is required, use a supported format (e.g. bitsandnames/awq via the library's own quant path)

Example fix

# before
--model /models/flux-dev-Q4_K_M.gguf
# after
--model black-forest-labs/FLUX.1-dev

from pathlib import Path
is_gguf = any(p.suffix == '.gguf' for p in Path(model_path).glob('*.gguf')) if Path(model_path).is_dir() else model_path.endswith('.gguf')

try:
    cfg = get_hf_config(path, ...)
except NotImplementedError:
    cfg = get_hf_config(fallback_hf_path, ...)

When it happens

Trigger: Passing a GGUF-quantized model path (e.g. something quantized with llama.cpp tooling) to load_native, load_customized, or create_kt_config_from_server_args, all of which route through get_hf_config.

Common situations: User points --model at a local GGUF file or a repo containing *.gguf weights to save memory; mixing llama.cpp artifacts into a diffusers/HF pipeline workflow.

Related errors

• GGUFConfig must be constructed from a GGUF checkpoint
• A GGUF encoder checkpoint cannot be combined with a second q
• Cannot parse checkpoint quantization for {component_name!r}:
• GGUF tensor {tensor.name} declares original shape {logical_s
• GGUF tensor {tensor.name} is quantized, but diffusion GGUF c

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f568004d72e8a840. Report an issue: GitHub.