sgl-project/sglang · error · ValueError

GGUF diffusion checkpoints require CUDA; the GGML kernels ha

Error message

GGUF diffusion checkpoints require CUDA; the GGML kernels have no {current_platform.device_type} implementation.

What it means

The GGML dequantization kernels used to load GGUF diffusion checkpoints only have CUDA implementations, so loading a GGUF transformer on any non-CUDA platform is rejected up front.

Source

Thrown at python/sglang/multimodal_gen/runtime/loader/transformer_load_utils.py:546

        )
    if server_args.quantization is not None:
        raise ValueError(
            f"--quantization {server_args.quantization} cannot be combined with "
            "a GGUF transformer, whose quantization is fixed by the checkpoint. "
            "Drop the flag, or use an unquantized checkpoint to quantize online."
        )
    # Nunchaku shares --transformer-weights-path with GGUF, and the GGUF plan is
    # resolved first, so without this the SVDQuant request would be dropped in
    # silence rather than refused.
    if server_args.nunchaku_config is not None:
        raise ValueError(
            "--enable-svdquant cannot be combined with a GGUF transformer: both "
            "supply the transformer weights. Point "
            "--transformer-weights-path at either an SVDQuant checkpoint or a "
            ".gguf, not one while requesting the other."
        )
    if not current_platform.is_cuda():
        raise ValueError(
            "GGUF diffusion checkpoints require CUDA; the GGML kernels have no "
            f"{current_platform.device_type} implementation."
        )
    uses_fsdp = (
        server_args.should_use_fsdp_for_component(component_name)
        if component_name is not None
        else server_args.use_fsdp_inference
    )
    if uses_fsdp:
        raise ValueError(
            "GGUF diffusion checkpoints are incompatible with FSDP inference. "
            "Run without --use-fsdp-inference, or keep this component offloaded "
            "so FSDP does not manage it."
        )
    if server_args.lora_path is not None:
        raise ValueError(
            "LoRA is not supported on a GGUF transformer: an adapter cannot be "
            "merged into packed GGML blocks. Use the unquantized checkpoint to "

View on GitHub (pinned to 0132848349)

Solutions

1. Run on an NVIDIA CUDA machine
2. Otherwise use a safetensors checkpoint compatible with your platform instead of GGUF

import torch
if not torch.cuda.is_available() and str(server_args.transformer_weights_path or '').endswith('.gguf'):
    raise SystemExit('GGUF transformer requires CUDA')

When it happens

Trigger: current_platform.is_cuda() is False (e.g. ROCm/HIP, CPU, Metal) while a .gguf transformer is requested via --transformer-weights-path.

Common situations: Running on AMD GPUs without CUDA support, CPU-only dev boxes, or a container without a visible CUDA device.

Related errors

• Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `
• {name}_block_cnt and {name}_block_idx must be on the same de
• {name}_block tensors must live on CUDA
• {name} must live on CUDA
• SplitKV partial output (mO) must be Float32

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/061e1ce5c21e7fe2. Report an issue: GitHub.