vllm-project/vllm · error · ValueError

Fast prefill optimization for KV sharing is not compatible w

Error message

Fast prefill optimization for KV sharing is not compatible with EAGLE as EAGLE requires correct logits for all tokens while fast prefill gives incorrect logits for prompt tokens.

What it means

`--kv-sharing-fast-prefill` skips full attention computation for prompt tokens whose KV is shared, which produces incorrect logits for those tokens. EAGLE speculative decoding needs correct logits for every token to build its draft distribution, so the combination is rejected at validation time.

Source

Thrown at vllm/config/vllm.py:1550

                logger.info_once("Cudagraph is disabled under eager mode")
                self.compilation_config.cudagraph_mode = CUDAGraphMode.NONE
                # override related settings when enforce eager
                self.compilation_config.max_cudagraph_capture_size = 0
                self.compilation_config.cudagraph_capture_sizes = []
            else:
                self.compilation_config.cudagraph_num_of_warmups = 1

            self._set_cudagraph_sizes()

        else:
            self.compilation_config.cudagraph_mode = CUDAGraphMode.NONE

        if self.cache_config.kv_sharing_fast_prefill:
            if (
                self.speculative_config is not None
                and self.speculative_config.use_eagle()
            ):
                raise ValueError(
                    "Fast prefill optimization for KV sharing is not "
                    "compatible with EAGLE as EAGLE requires correct logits "
                    "for all tokens while fast prefill gives incorrect logits "
                    "for prompt tokens."
                )

            logger.warning_once(
                "--kv-sharing-fast-prefill requires changes on model side for "
                "correctness and to realize prefill savings."
            )

        if (
            self.model_config
            and self.model_config.architecture == "WhisperForConditionalGeneration"
            and os.environ.get("VLLM_WORKER_MULTIPROC_METHOD") != "spawn"
        ):
            logger.warning_once(
                "Whisper is known to have issues with "

View on GitHub (pinned to c794754062)

Solutions

1. Remove `--kv-sharing-fast-prefill` when using EAGLE speculative decoding.
2. Or use a non-EAGLE spec-decode method (e.g. ngram GPU) that tolerates incorrect prompt logits.
3. Or disable speculative decoding entirely for this deployment.

Example fix

# before
vllm serve model --kv-sharing-fast-prefill \
  --speculative-config '{"method":"eagle",...}'

# after
vllm serve model \
  --speculative-config '{"method":"eagle",...}'

if kv_sharing_fast_prefill and spec_config and spec_config.get("method", "").lower().startswith(("eagle", "mtp")):
    kv_sharing_fast_prefill = False

try:
    LLM(**args)
except ValueError as e:
    if "Fast prefill optimization for KV sharing" in str(e):
        args["kv_sharing_fast_prefill"] = False
    else:
        raise

When it happens

Trigger: Launching with `--kv-sharing-fast-prefill` and a speculative config that uses EAGLE (`speculative_config.use_eagle()` true, incl. MTP variants).

Common situations: Users of models with cross-layer KV sharing (e.g. MLA/CLV models like some Qwen3/Gemma variants) enabling the fast-prefill latency optimization while also running EAGLE drafts.

Related errors

• Unknown KV cache group kind '{kind}' in backend_per_kind. Va
• offload_num_in_group ({self.prefetch.offload_num_in_group})
• rejection_sample_method='synthetic' requires exactly one of
• synthetic_acceptance_rates must have length {n}, got {rates}
• synthetic_acceptance_rates entries must be in [0, 1], got {r

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/1bf39668d7bb5814. Report an issue: GitHub.