sgl-project/sglang · error · NotImplementedError

CPU offloading is unsupported under the page-major layout (T

Error message

CPU offloading is unsupported under the page-major layout (TODO: split token ids into page/slot for the 4-D index).

What it means

CPU offloading (get_cpu_copy) requires copying per-layer contiguous KV slices indexed token-major, but the page-major 4-D layout interleaves layers within pages, so the inherited implementation would copy wrong bytes. The method raises NotImplementedError until a page/slot-splitting index is implemented.

Source

Thrown at python/sglang/srt/mem_cache/memory_pool.py:3320

            tgt_loc,
            src_loc,
            page_size=self.page_size,
        )

    # The methods below assume the per-layer contiguous 3-D layout. The 4-D
    # strided envelope views have no per-layer contiguous region (their bytes are
    # interleaved layer-major within each page) and index page-major, not
    # token-major. Inheriting them would silently mis-index; fail loudly instead.

    def get_contiguous_buf_infos(self):
        raise NotImplementedError(
            "page-major layout has no per-layer contiguous regions; KV transfer / "
            "disaggregation is unsupported (TODO: expose the single _raw buffer "
            "with a page-aware transfer scheme)."
        )

    def get_cpu_copy(self, indices, mamba_indices=None):
        raise NotImplementedError(
            "CPU offloading is unsupported under the page-major layout "
            "(TODO: split token ids into page/slot for the 4-D index)."
        )

    def load_cpu_copy(self, kv_cache_cpu, indices, mamba_indices=None):
        raise NotImplementedError(
            "CPU offloading is unsupported under the page-major layout "
            "(TODO: split token ids into page/slot for the 4-D index)."
        )

    def set_kv_buffer_prefix_valid(self, *args, **kwargs):
        raise NotImplementedError(
            "prefix-valid commit is unsupported under the page-major layout "
            "(_set_kv_buffer_prefix_valid_impl assumes 3-D contiguous + row_dim)."
        )


class MHATokenToKVPoolMXFP8(MHATokenToKVPool):

View on GitHub (pinned to 0132848349)

Solutions

Disable CPU offloading / hierarchical cache when using the page-major layout
Switch the pool to the token-major 3-D layout (page_size=1) to keep CPU offloading usable
Implement page/slot decomposition for the 4-D index before calling get_cpu_copy

Defensive patterns

Strategy: validation

Validate before calling

def offload_supported(pool) -> bool:
    return not getattr(pool, 'page_size', 1) > 1  # token-major 3-D pools only

Try / catch

try:
    cpu_copy = pool.get_cpu_copy(indices)
except NotImplementedError:
    logger.warning('CPU offload unavailable for this layout; skipping offload')

Prevention

Validate enable-hierarchical-cache/offload flags against pool layout at launch
Keep a compatibility matrix of layout vs offload features in CI
Fail fast at startup instead of at first offload attempt

When it happens

Trigger: Allocating the page-major KV pool and enabling CPU offloading of KV cache (hi-CPU offload / hierarchical cache), which calls pool.get_cpu_copy(indices, mamba_indices).

Common situations: Combining a paged page-major KV layout config with --enable-hierarchical-cache or CPU offload settings; recent versions introducing the page-major envelope layout while offload code still assumes 3-D.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/16306fca9be32d72. Report an issue: GitHub.