sgl-project/sglang · error · ValueError

--prefill-only-disable-kv-cache currently requires the FA pr

Error message

--prefill-only-disable-kv-cache currently requires the FA prefill backend (fa3/fa4), but got prefill backend {prefill_backend!r}. Other prefill-only workloads and backends may be supported in a future change.

What it means

The fa_skip_kv_cache fast path in SGLang is implemented only in the FlashAttention prefill backends. _handle_prefill_only_disable_kv_cache runs after attention backend resolution and raises if the resolved prefill backend is not fa3 or fa4.

Source

Thrown at python/sglang/srt/server_args.py:8238

        the default attention_backend if unset) and _handle_multi_item_scoring()
        (which may further mutate it). The assertion below guards against
        accidental call-site reordering: if the resolved attention_backend is
        still None, backends haven't settled yet and the resolved (prefill,
        decode) pair would be a stale (None, None).
        """
        cfg = resolving_view(self)

        if not cfg.prefill_only_disable_kv_cache:
            return

        assert resolved_view(self).attention_backend is not None, (
            "_handle_prefill_only_disable_kv_cache must run after "
            "_handle_attention_backend_compatibility() so the prefill backend is resolved."
        )

        prefill_backend, _ = self._resolved_attention_backends()
        if prefill_backend not in ("fa3", "fa4"):
            raise ValueError(
                "--prefill-only-disable-kv-cache currently requires the FA prefill backend "
                f"(fa3/fa4), but got prefill backend {prefill_backend!r}. Other prefill-only "
                "workloads and backends may be supported in a future change."
            )

    def _handle_hicache_ratio_default(self):
        """Default the host/device ratio per host memory mode.

        Runs before the dummy-model boundary: direct HostKVCache consumers
        (unit fixtures, dummy-model launches) must never see a None ratio.
        buffer_only stages in flight rather than retaining, so it needs only
        enough to cover the write backlog plus parked prefetches.

        A decode server keeps the ratio unset here: kv_cache_builder resolves
        it against the retraction-backup backend (1.0 for host_pool, else 2.0).
        """
        cfg = resolving_view(self)
        if cfg.hicache_ratio is None and cfg.disaggregation_mode != "decode":

View on GitHub (pinned to 0132848349)

Solutions

  1. Set the prefill backend to FA3/FA4: --attention-backend fa3 (requires supported Hopper/Blackwell GPU + CUDA build)
  2. Remove --prefill-only-disable-kv-cache if you must use another backend (e.g. flashinfer on other hardware)
  3. On unsupported hardware, keep the real KV pool and use quantization instead

Example fix

# before
--prefill-only-disable-kv-cache --attention-backend flashinfer
# after
--prefill-only-disable-kv-cache --attention-backend fa3
Defensive patterns

Strategy: validation

Validate before calling

FA_BACKENDS = {"fa3", "fa4"}
if want_disable_kv_cache and attention_backend not in FA_BACKENDS:
    raise SystemExit("--prefill-only-disable-kv-cache needs --attention-backend fa3 on Hopper/Blackwell")

Prevention

When it happens

Trigger: Launching with --prefill-only-disable-kv-cache while the resolved prefill attention backend is anything other than fa3/fa4 (e.g. flashinfer, triton, torch_native, or an AMD backend), typically via --attention-backend or a hardware default.

Common situations: Running on non-Hopper/Blackwell hardware where FA3 is unavailable, or a config that explicitly sets --attention-backend flashinfer for throughput, then adding the KV-cache-disabling flag.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/da3c4a85783d2827. Report an issue: GitHub.