sgl-project/sglang · error · UnsupportedQuantForIPCError

[weight_cache:{where}] quantization method {quant_method!r}

Error message

[weight_cache:{where}] quantization method {quant_method!r} is not verified for CUDA IPC zero-copy weight sharing. Its process_weights_after_loading may stamp Python-side metadata (e.g. format_ue8m0) or repack/transpose weights into shapes the meta-initialized client cannot reproduce, which would silently serve wrong-numerics weights. Verified methods: {verified}. Note: FP8 is only verified for block-wise configs (weight_block_size set), not per-tensor FP8. Disable the weight cache (--weight-cache-mode off)

What it means

Raised by check_ipc_quant_support when the model's quantization method is not in the IPC allowlist. Zero-copy CUDA IPC sharing is only verified for specific quants (block-wise FP8 etc.); unverified process_weights_after_loading may produce wrong-numerics weights in the meta-initialized client, so the loader refuses.

Source

Thrown at python/sglang/srt/weight_cache/protocol.py:189

    if predicate is None:
        return False
    return bool(predicate(quant_config))


def check_ipc_quant_support(
    quant_method: str, quant_config: Any, *, where: str
) -> None:
    """Hard-error unless `quant_method` is verified safe for IPC zero-copy sharing.

    `where` is a short tag (e.g. "daemon"/"client") used only in the error
    message. Raises UnsupportedQuantForIPCError with an actionable message.
    """
    if is_ipc_quant_supported(quant_method, quant_config):
        return
    verified = ", ".join(
        (repr(m) if m else "'' (unquantized)") for m in IPC_QUANT_ALLOWLIST
    )
    raise UnsupportedQuantForIPCError(
        f"[weight_cache:{where}] quantization method {quant_method!r} is not "
        f"verified for CUDA IPC zero-copy weight sharing. Its "
        f"process_weights_after_loading may stamp Python-side metadata "
        f"(e.g. format_ue8m0) or repack/transpose weights into shapes the "
        f"meta-initialized client cannot reproduce, which would silently serve "
        f"wrong-numerics weights. Verified methods: {verified}. Note: FP8 is "
        f"only verified for block-wise configs (weight_block_size set), not "
        f"per-tensor FP8. Disable the weight cache (--weight-cache-mode off) "
        f"for this model."
    )


# ---------------------------------------------------------------------------
# Socket protocol helpers
# ---------------------------------------------------------------------------


MAX_MSG_SIZE = 256 * 1024 * 1024  # 256 MiB

View on GitHub (pinned to 0132848349)

Solutions

1. Run with --weight-cache-mode off to load weights normally
2. Switch to a verified quant config (e.g. FP8 with weight_block_size set)
3. Request/verify support for the quant method upstream before enabling IPC sharing

Example fix

# before
server_args.weight_cache_mode = "ipc"
# after
server_args.weight_cache_mode = "off"

from sglang.srt.weight_cache.protocol import is_ipc_quant_supported
if weight_cache_on and not is_ipc_quant_supported(quant_method, quant_config):
    server_args.weight_cache_mode = 'off'

When it happens

Trigger: Launching with --weight-cache-mode ipc (or on) using per-tensor FP8, unverified quant methods (e.g. some AWQ/GPTQ builds), or '' (unquantized) when not allowlisted; called from load/load_model.

Common situations: Enabling weight cache on a model checkpoint quantized with a method outside IPC_QUANT_ALLOWLIST; switching a model from block-wise FP8 to per-tensor FP8 without changing cache settings.

Related errors

• The quantization method `{quantization}` is already exists.
• The quantization config must be a subclass of `QuantizationC
• Invalid quantization method: {quantization}
• kitchen_w4a4 is inferred from per-layer checkpoint metadata;
• GGUFConfig must be constructed from a GGUF checkpoint

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3240b402b0844988. Report an issue: GitHub.