sgl-project/sglang · error · ValueError

GGUF diffusion checkpoints are incompatible with FSDP infere

Error message

GGUF diffusion checkpoints are incompatible with FSDP inference. Run without --use-fsdp-inference, or keep this component offloaded so FSDP does not manage it.

What it means

GGUF diffusion checkpoints cannot be sharded/managed by FSDP because the packed GGML blocks don't fit FSDP's parameter expectations, so the combination is refused.

Source

Thrown at python/sglang/multimodal_gen/runtime/loader/transformer_load_utils.py:556

    if server_args.nunchaku_config is not None:
        raise ValueError(
            "--enable-svdquant cannot be combined with a GGUF transformer: both "
            "supply the transformer weights. Point "
            "--transformer-weights-path at either an SVDQuant checkpoint or a "
            ".gguf, not one while requesting the other."
        )
    if not current_platform.is_cuda():
        raise ValueError(
            "GGUF diffusion checkpoints require CUDA; the GGML kernels have no "
            f"{current_platform.device_type} implementation."
        )
    uses_fsdp = (
        server_args.should_use_fsdp_for_component(component_name)
        if component_name is not None
        else server_args.use_fsdp_inference
    )
    if uses_fsdp:
        raise ValueError(
            "GGUF diffusion checkpoints are incompatible with FSDP inference. "
            "Run without --use-fsdp-inference, or keep this component offloaded "
            "so FSDP does not manage it."
        )
    if server_args.lora_path is not None:
        raise ValueError(
            "LoRA is not supported on a GGUF transformer: an adapter cannot be "
            "merged into packed GGML blocks. Use the unquantized checkpoint to "
            "serve LoRA."
        )
    # H3's AdaLN paths read the transformer's safetensors directly -- the cache
    # builder needs unquantized weights, and the online rebuild is handed the
    # safetensors file list, which is empty for a GGUF load.
    if server_args.minimax_h3_adaln_online:
        raise ValueError(
            "--minimax-h3-adaln-online rebuilds AdaLN outputs from the "
            "safetensors checkpoint and cannot read a GGUF transformer."
        )

View on GitHub (pinned to 0132848349)

Solutions

Run without --use-fsdp-inference for this GGUF model
Keep the transformer component offloaded/excluded so FSDP does not manage it
Use an FSDP-compatible safetensors checkpoint if FSDP is required

Example fix

# before
--use-fsdp-inference --transformer-weights-path model.gguf
# after
--transformer-weights-path model.gguf

Defensive patterns

Strategy: validation

Validate before calling

if str(server_args.transformer_weights_path or '').endswith('.gguf') and (server_args.use_fsdp_inference or server_args.should_use_fsdp_for_component('transformer')):
    raise SystemExit('GGUF transformer is incompatible with FSDP inference')

Prevention

Don't enable --use-fsdp-inference with GGUF checkpoints
Offload the transformer component so FSDP skips it

When it happens

Trigger: server_args.use_fsdp_inference is True, or should_use_fsdp_for_component(component_name) returns True for the component being loaded, while loading a GGUF transformer.

Common situations: Enabling --use-fsdp-inference for multi-GPU memory sharding and then pointing at a .gguf checkpoint; or FSDP picking up the transformer component because it wasn't offloaded/excluded.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d41d82ceaf01a3d8. Report an issue: GitHub.