sgl-project/sglang · error · ValueError

--prefill-only-disable-kv-cache does not currently support -

Error message

--prefill-only-disable-kv-cache does not currently support --kv-cache-dtype=nvfp4 or --kv-cache-dtype=fp4_mx_block16 because the FP4 pool uses a separate allocation path.

What it means

The no-op KV pool used by --prefill-only-disable-kv-cache assumes the standard token-to-KV allocation path. FP4 KV cache dtypes (nvfp4, fp4_mx_block16) allocate the pool through a separate FP4 path, so the combination is rejected up front in _validate_prefill_only_disable_kv_cache_args.

Source

Thrown at python/sglang/srt/server_args.py:8162

        Backend resolution is checked separately by
        _handle_prefill_only_disable_kv_cache after backends settle.
        """
        cfg = resolving_view(self)
        if not cfg.prefill_only_disable_kv_cache:
            return

        # This flag is intentionally scoped to embedding mode for now. Other
        # prefill-only paths (for example scoring and MIS) can benefit from
        # the same idea later, but some of them still stage K/V through the
        # paged cache today.
        if not cfg.is_embedding:
            raise ValueError(
                "--prefill-only-disable-kv-cache currently requires --is-embedding. "
                "Other prefill-only workloads may be supported in a future change once "
                "their attention paths stop reading or writing the paged KV cache."
            )
        if cfg.kv_cache_dtype in ("nvfp4", "fp4_mx_block16"):
            raise ValueError(
                "--prefill-only-disable-kv-cache does not currently support "
                "--kv-cache-dtype=nvfp4 or --kv-cache-dtype=fp4_mx_block16 because "
                "the FP4 pool uses a separate allocation path."
            )
        if cfg.kv_cache_dtype == "mxfp8":
            raise ValueError(
                "--prefill-only-disable-kv-cache does not currently support "
                "--kv-cache-dtype=mxfp8 because the MXFP8 pool stores separate "
                "scale-factor buffers."
            )

        # Structural preconditions for the FA backend's fa_skip_kv_cache path,
        # which is the only embedding path that doesn't read or write the pool:
        # - chunked_prefill_size == -1 keeps a request in a single forward,
        #   so K/V never has to be reused across prefill chunks.
        # - disable_radix_cache stops the prefix cache from indexing pool
        #   slots that no longer hold real data.
        if cfg.chunked_prefill_size != -1:

View on GitHub (pinned to 0132848349)

Solutions

Remove --kv-cache-dtype=nvfp4 / fp4_mx_block16 (fall back to the default dtype) when using --prefill-only-disable-kv-cache
If FP4 KV cache is required, drop --prefill-only-disable-kv-cache and rely on quantization for memory savings
Wait for upstream support of the FP4 allocation path with the no-op pool

Example fix

# before
--is-embedding --prefill-only-disable-kv-cache --kv-cache-dtype nvfp4
# after
--is-embedding --prefill-only-disable-kv-cache

Defensive patterns

Strategy: validation

Validate before calling

FP4_DTYPES = {"nvfp4", "fp4_mx_block16"}
if want_disable_kv_cache and kv_cache_dtype in FP4_DTYPES:
    raise SystemExit("pick one: --prefill-only-disable-kv-cache or FP4 kv-cache-dtype")

Prevention

Keep a compatibility matrix of kv-cache-dtype vs memory flags in your serving config repo
Never stack KV memory optimizations (quantized pool + disabled pool) without checking the validator

When it happens

Trigger: Launching with --prefill-only-disable-kv-cache together with --kv-cache-dtype nvfp4 or --kv-cache-dtype fp4_mx_block16 (cfg.kv_cache_dtype in ("nvfp4","fp4_mx_block16")).

Common situations: A user enables FP4 KV cache quantization to save memory and separately enables KV-cache disabling for embedding serving, not realizing the FP4 pool has its own allocator.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/784f2974b8937419. Report an issue: GitHub.