{"record":{"id":"cbf1081ed773ecaf","repo":"Comfy-Org/ComfyUI","slug":"gemma4-chunked-prefill-past-the-sliding-window-is","errorCode":null,"errorMessage":"gemma4: chunked prefill past the sliding window is not supported","messagePattern":"gemma4: chunked prefill past the sliding window is not supported","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"comfy/text_encoders/gemma4.py","lineNumber":252,"sourceCode":"\n                # prefill: attend the local sequence, persist the tail into the cache\n                capacity = fixed_cache.key.shape[2]\n                index = fixed_cache.index\n                if index + seq_length <= capacity:\n                    fixed_cache.key[:, :, index:index + seq_length] = xk\n                    fixed_cache.value[:, :, index:index + seq_length] = xv\n                    if index > 0:\n                        xk = fixed_cache.key[:, :, :index + seq_length]\n                        xv = fixed_cache.value[:, :, :index + seq_length]\n                elif index == 0:\n                    # prefill longer than the sliding ring: attend the full local K/V\n                    # (per-query windows come from the prefill sliding mask), cache only\n                    # the last `capacity` keys at their wrapped slots (position % capacity)\n                    slots = torch.arange(seq_length - capacity, seq_length, device=xk.device) % capacity\n                    fixed_cache.key.index_copy_(2, slots, xk[:, :, -capacity:])\n                    fixed_cache.value.index_copy_(2, slots, xv[:, :, -capacity:])\n                else:\n                    raise RuntimeError(\"gemma4: chunked prefill past the sliding window is not supported\")\n                present_key_value = fixed_cache\n            elif past_key_value is not None:\n                cumulative_len = 0\n                if len(past_key_value) > 0:\n                    past_key, past_value, cumulative_len = past_key_value\n                    xk = torch.cat((past_key, xk), dim=2)\n                    xv = torch.cat((past_value, xv), dim=2)\n                new_cumulative = cumulative_len + seq_length\n                if sliding_window is not None and xk.shape[2] > sliding_window - 1:\n                    cache_k = xk[:, :, -(sliding_window - 1):]\n                    cache_v = xv[:, :, -(sliding_window - 1):]\n                else:\n                    cache_k = xk\n                    cache_v = xv\n                present_key_value = (cache_k, cache_v, new_cumulative)\n\n            # KV for sharing: full xk/xv that SDPA sees (not evicted cache)\n            shareable_kv = (xk, xv)","sourceCodeStart":234,"sourceCodeEnd":270,"githubUrl":"https://github.com/Comfy-Org/ComfyUI/blob/1c6d8d45b3693bfbb32385b410d813a7fd6be216/comfy/text_encoders/gemma4.py#L234-L270","documentation":"The Gemma4 attention path in comfy/text_encoders/gemma4.py implements a fixed-size ring KV cache for sliding-window attention. Prefill is supported either chunk-by-chunk within the ring (index>0 short path) or as a single first chunk longer than the window (index==0 path that re-wraps the last `capacity` keys). Any chunk that starts past the window (index>0 with a long cumulative sequence) would need keys the ring no longer holds, so it raises instead of silently attending to wrong context.","triggerScenarios":"Feeding Gemma4 extremely long prompts in multiple prefill chunks such that a later chunk begins at an index beyond the sliding window/ring capacity; custom encode loops that pass incremental chunks plus a fixed_cache; context lengths exceeding the configured cache capacity with chunked scheduling.","commonSituations":"Very long multimodal prompts (many images plus long text) overflowing the fixed cache; custom nodes that chunk prompt encoding manually; models whose sliding_window config is smaller than the actual prompt length combined with chunked prefill.","solutions":["Encode the whole long prompt as a single initial chunk (index==0 path) instead of feeding multiple chunks past the window.","Shorten the prompt / reduce number of images so total tokens fit within the sliding-window cache capacity.","If you control the cache, increase the fixed_cache capacity so the ring covers the full prefill length."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"total = sum(chunk_len for chunk_len in planned_chunks)\nassert total <= fixed_cache_capacity, 'prompt exceeds sliding-window cache; encode as one chunk or shorten'","typeGuard":null,"tryCatchPattern":"try:\n    emb = encoder.encode(chunks)\nexcept RuntimeError as e:\n    if 'chunked prefill' in str(e):\n        emb = encoder.encode([all_tokens])  # single-chunk prefill path\n    else:\n        raise","preventionTips":["Send the whole long prompt in one encode call instead of manual chunking.","Keep multimodal prompts within the model's sliding-window capacity (fewer images / shorter text)."],"tags":["text-encoder","gemma4","sliding-window","kv-cache","long-context"],"backgroundTag":null,"analyzedSha":"1c6d8d45b3693bfbb32385b410d813a7fd6be216","analyzedAt":"2026-08-14T19:37:18.893Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}