sgl-project/sglang · error · ValueError

--prefill-only-disable-kv-cache currently requires --is-embe

Error message

--prefill-only-disable-kv-cache currently requires --is-embedding. Other prefill-only workloads may be supported in a future change once their attention paths stop reading or writing the paged KV cache.

What it means

SGLang's --prefill-only-disable-kv-cache flag replaces the paged KV cache with a no-op MHA pool, and that path is currently gated to embedding serving (--is-embedding). Other prefill-only workloads (scoring, MIS) still stage K/V through the paged cache, so enabling the flag without embedding mode is rejected during server-args resolution.

Source

Thrown at python/sglang/srt/server_args.py:8156

        run_post_process_pass(self, _pipeline_parallel_overlap_disable)

    def _validate_prefill_only_disable_kv_cache_args(self):
        """Validate --prefill-only-disable-kv-cache flag/precondition constraints.

        Backend resolution is checked separately by
        _handle_prefill_only_disable_kv_cache after backends settle.
        """
        cfg = resolving_view(self)
        if not cfg.prefill_only_disable_kv_cache:
            return

        # This flag is intentionally scoped to embedding mode for now. Other
        # prefill-only paths (for example scoring and MIS) can benefit from
        # the same idea later, but some of them still stage K/V through the
        # paged cache today.
        if not cfg.is_embedding:
            raise ValueError(
                "--prefill-only-disable-kv-cache currently requires --is-embedding. "
                "Other prefill-only workloads may be supported in a future change once "
                "their attention paths stop reading or writing the paged KV cache."
            )
        if cfg.kv_cache_dtype in ("nvfp4", "fp4_mx_block16"):
            raise ValueError(
                "--prefill-only-disable-kv-cache does not currently support "
                "--kv-cache-dtype=nvfp4 or --kv-cache-dtype=fp4_mx_block16 because "
                "the FP4 pool uses a separate allocation path."
            )
        if cfg.kv_cache_dtype == "mxfp8":
            raise ValueError(
                "--prefill-only-disable-kv-cache does not currently support "
                "--kv-cache-dtype=mxfp8 because the MXFP8 pool stores separate "
                "scale-factor buffers."
            )

        # Structural preconditions for the FA backend's fa_skip_kv_cache path,

View on GitHub (pinned to 0132848349)

Solutions

  1. Add --is-embedding to the launch command (or use an embedding model so it is auto-set)
  2. If the workload is not embedding (scoring, MIS), remove --prefill-only-disable-kv-cache until support lands
  3. Track the upstream change that makes other prefill-only attention paths pool-free before re-enabling

Example fix

# before
python -m sglang.launch_server --model qwen3-reranker --prefill-only-disable-kv-cache
# after
python -m sglang.launch_server --model Qwen3-Embedding --is-embedding --prefill-only-disable-kv-cache
Defensive patterns

Strategy: validation

Validate before calling

from sglang.srt.server_args import prepare_server_args
args = ["--model", MODEL, *user_flags]
if "--prefill-only-disable-kv-cache" in args and "--is-embedding" not in args and not is_embedding_model(MODEL):
    raise SystemExit("--prefill-only-disable-kv-cache requires --is-embedding")
server_args = prepare_server_args(args)

Prevention

When it happens

Trigger: Launching the server with --prefill-only-disable-kv-cache but without --is-embedding (or without using an embedding model that auto-sets it), causing _validate_prefill_only_disable_kv_cache_args to raise during the resolution pipeline.

Common situations: A user tries to shave memory off a non-embedding prefill workload (e.g. reranking/scoring) by disabling the KV cache, assuming the flag is generic; or an embedding workload where --is-embedding was not passed/auto-detected.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/eb1529fbdaa68b5a. Report an issue: GitHub.