Comfy-Org/ComfyUI · error · RuntimeError

gemma4: chunked prefill past the sliding window is not suppo

Error message

gemma4: chunked prefill past the sliding window is not supported

What it means

The Gemma4 attention path in comfy/text_encoders/gemma4.py implements a fixed-size ring KV cache for sliding-window attention. Prefill is supported either chunk-by-chunk within the ring (index>0 short path) or as a single first chunk longer than the window (index==0 path that re-wraps the last `capacity` keys). Any chunk that starts past the window (index>0 with a long cumulative sequence) would need keys the ring no longer holds, so it raises instead of silently attending to wrong context.

Source

Thrown at comfy/text_encoders/gemma4.py:252

                # prefill: attend the local sequence, persist the tail into the cache
                capacity = fixed_cache.key.shape[2]
                index = fixed_cache.index
                if index + seq_length <= capacity:
                    fixed_cache.key[:, :, index:index + seq_length] = xk
                    fixed_cache.value[:, :, index:index + seq_length] = xv
                    if index > 0:
                        xk = fixed_cache.key[:, :, :index + seq_length]
                        xv = fixed_cache.value[:, :, :index + seq_length]
                elif index == 0:
                    # prefill longer than the sliding ring: attend the full local K/V
                    # (per-query windows come from the prefill sliding mask), cache only
                    # the last `capacity` keys at their wrapped slots (position % capacity)
                    slots = torch.arange(seq_length - capacity, seq_length, device=xk.device) % capacity
                    fixed_cache.key.index_copy_(2, slots, xk[:, :, -capacity:])
                    fixed_cache.value.index_copy_(2, slots, xv[:, :, -capacity:])
                else:
                    raise RuntimeError("gemma4: chunked prefill past the sliding window is not supported")
                present_key_value = fixed_cache
            elif past_key_value is not None:
                cumulative_len = 0
                if len(past_key_value) > 0:
                    past_key, past_value, cumulative_len = past_key_value
                    xk = torch.cat((past_key, xk), dim=2)
                    xv = torch.cat((past_value, xv), dim=2)
                new_cumulative = cumulative_len + seq_length
                if sliding_window is not None and xk.shape[2] > sliding_window - 1:
                    cache_k = xk[:, :, -(sliding_window - 1):]
                    cache_v = xv[:, :, -(sliding_window - 1):]
                else:
                    cache_k = xk
                    cache_v = xv
                present_key_value = (cache_k, cache_v, new_cumulative)

            # KV for sharing: full xk/xv that SDPA sees (not evicted cache)
            shareable_kv = (xk, xv)

View on GitHub (pinned to 1c6d8d45b3)

Solutions

1. Encode the whole long prompt as a single initial chunk (index==0 path) instead of feeding multiple chunks past the window.
2. Shorten the prompt / reduce number of images so total tokens fit within the sliding-window cache capacity.
3. If you control the cache, increase the fixed_cache capacity so the ring covers the full prefill length.

total = sum(chunk_len for chunk_len in planned_chunks)
assert total <= fixed_cache_capacity, 'prompt exceeds sliding-window cache; encode as one chunk or shorten'

try:
    emb = encoder.encode(chunks)
except RuntimeError as e:
    if 'chunked prefill' in str(e):
        emb = encoder.encode([all_tokens])  # single-chunk prefill path
    else:
        raise

When it happens

Trigger: Feeding Gemma4 extremely long prompts in multiple prefill chunks such that a later chunk begins at an index beyond the sliding window/ring capacity; custom encode loops that pass incremental chunks plus a fixed_cache; context lengths exceeding the configured cache capacity with chunked scheduling.

Common situations: Very long multimodal prompts (many images plus long text) overflowing the fixed cache; custom nodes that chunk prompt encoding manually; models whose sliding_window config is smaller than the actual prompt length combined with chunked prefill.

Related errors

• Attempting to resize to a 0 x 0 image. Resized height should
• Resizing [{height}x{width}] to [{target_height}x{target_widt
• Krea2 expects conditioning with {self.txtlayers}x{self.txtdi
• Unexpected type for duration key, must be str, int or float
• Lens tokenizer requires the ``tokenizer_json`` byte tensor i

AI-assisted analysis of Comfy-Org/ComfyUI@1c6d8d45b3 (2026-08-14). Data as JSON: /api/errors/cbf1081ed773ecaf. Report an issue: GitHub.