{"record":{"id":"a95989c29921c09b","repo":"sgl-project/sglang","slug":"batcheddecodecontext-requires-full-kv-pool-index-b","errorCode":null,"errorMessage":"BatchedDecodeContext requires full_kv_pool_index_by_layer when the fused AOT RoPE + pool-scatter kernel is active","messagePattern":"BatchedDecodeContext requires full_kv_pool_index_by_layer when the fused AOT RoPE \\+ pool-scatter kernel is active","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/hardware_backend/mlx/kv_cache/attention_wrapper.py","lineNumber":82,"sourceCode":"\n    def __post_init__(self) -> None:\n        seq_lens = self.seq_lens\n        max_seq_len = max(seq_lens)\n        self.offsets = mx.array(seq_lens, dtype=mx.int32)\n        self.max_len = max_seq_len + 1\n        self.valid_lens = self.offsets + 1\n        self.needs_padding = min(seq_lens) < max_seq_len\n        self.pad_sizes = [max_seq_len - s for s in seq_lens]\n        self.positions = mx.arange(self.max_len) if self.needs_padding else None\n        if not self.attention_pool_index_by_layer:\n            self.attention_pool_index_by_layer = {\n                idx: idx for idx in range(len(self.attention_layer_caches))\n            }\n        if self.aot.rope is not None and not self.full_kv_pool_index_by_layer:\n            # The fused scatter addresses pool buffers by full-attention index;\n            # defaulting to the cache index would write the wrong buffer\n            # whenever sliding-window layers are interleaved.\n            raise ValueError(\n                \"BatchedDecodeContext requires full_kv_pool_index_by_layer \"\n                \"when the fused AOT RoPE + pool-scatter kernel is active\"\n            )\n\n    def decode_padding(\n        self, window: int | None\n    ) -> tuple[list[int], Optional[mx.array]]:\n        \"\"\"Right-pad sizes and the keep-mask for one decode step.\n\n        Requests are padded to a common KV width so they can be batched into\n        one SDPA call.  Without a window that width is ``max_len``; a\n        sliding-window layer only reads the trailing ``window`` keys, so its\n        width is ``max(min(seq_len + 1, window))`` instead -- which is why the\n        context's full-length metadata cannot be reused for it.\n\n        The mask is boolean (``True`` keeps the key), broadcast-shaped\n        ``(B, 1, 1, width)``, and ``None`` when no request needs padding.\n        \"\"\"","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/hardware_backend/mlx/kv_cache/attention_wrapper.py#L64-L100","documentation":"When the fused AOT RoPE + pool-scatter kernel is enabled (`self.aot.rope is not None`), the scatter writes into full-attention pool buffers, so the decode context must carry a `full_kv_pool_index_by_layer` mapping. Without it the kernel would fall back to per-cache indices and write to the wrong buffers when sliding-window layers are interleaved with full-attention layers. The dataclass's __post_init__ therefore validates this invariant.","triggerScenarios":"Constructing `BatchedDecodeContext` with `aot.rope` set but leaving `full_kv_pool_index_by_layer` empty/None — typically when a caller builds the context manually or a code path forgets to populate the layer→pool-index map for hybrid-window models.","commonSituations":"Enabling SGLANG_MLX_USE_CUSTOM_ROPE / the AOT fused kernel on a model with interleaved sliding-window layers; upgrading SGLang where BatchedDecodeContext gained a new required field; custom decode loops constructing the context directly.","solutions":["Populate `full_kv_pool_index_by_layer` (layer index -> full-attention KV pool index) whenever building the context with the fused AOT kernel active.","If the fused kernel isn't needed, disable it (unset SGLANG_MLX_USE_CUSTOM_ROPE) so `aot.rope` is None.","Update third-party/copy-pasted BatchedDecodeContext constructions to the new required field after upgrades."],"exampleFix":"# before\nctx = BatchedDecodeContext(aot=aot_with_rope, ...)\n\n# after\nctx = BatchedDecodeContext(\n    aot=aot_with_rope,\n    full_kv_pool_index_by_layer={i: pool.layout.full_pool_index(i) for i in attention_layers},\n    ...,\n)","handlingStrategy":"validation","validationCode":"if aot.rope is not None and not full_kv_pool_index_by_layer:\n    full_kv_pool_index_by_layer = build_full_pool_index_map(layout)\nctx = BatchedDecodeContext(aot=aot, full_kv_pool_index_by_layer=full_kv_pool_index_by_layer, ...)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Centralize BatchedDecodeContext construction in one factory that fills required fields when the AOT kernel is on.","Add a smoke test with the AOT kernel enabled plus a hybrid-window model."],"tags":["mlx","speculative-decode","aot-kernel","kv-cache","config-validation"],"backgroundTag":"missing-required-field","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}