sgl-project/sglang · critical · Exception

SWA Radix tree sanity check failed, ping @hanming-lu: {e}

Error message

SWA Radix tree sanity check failed, ping @hanming-lu: {e}

What it means

The SWA radix cache's internal sanity check (a series of asserts comparing evictable sizes and LRU list sizes) failed. This indicates an internal consistency bug in tree/LRU bookkeeping; the message asks to ping @hanming-lu because it is not expected to be user-fixable via config.

Source

Thrown at python/sglang/srt/mem_cache/swa_radix_cache.py:342

                assert (
                    x_lru.swa_lock_ref == 0
                ), f"x_lru should not be locked when idle, {x_lru.swa_lock_ref=}, {x_lru.swa_uuid=}, {x_lru.id=}"
                x_lru = getattr(x, self.prv)

            if self.is_swa_list:
                evictable_size = tree_cache.swa_evictable_size()
                lru_list_evictable_size = self.sanity_check_evictable_size()
            else:
                evictable_size = tree_cache.full_evictable_size()
                lru_list_evictable_size = self.sanity_check_evictable_size()

            assert (
                evictable_size == lru_list_evictable_size
            ), f"{self.is_swa_list=}, total nodes: {total_nodes}, total lru plus 1: {total_lru_plus_1}, evictable size: {evictable_size} != lru list evictable size: {lru_list_evictable_size}"
        except Exception as e:
            msg = f"SWA Radix tree sanity check failed, ping @hanming-lu: {e}"
            logger.error(msg)
            raise Exception(msg)


class SWARadixCache(BasePrefixCache):
    def __init__(self, params: CacheInitParams):
        assert isinstance(params.token_to_kv_pool_allocator, SWATokenToKVPoolAllocator)
        self.req_to_token_pool = params.req_to_token_pool
        self.token_to_kv_pool_allocator = params.token_to_kv_pool_allocator
        self.page_size = params.page_size
        self.disable = params.disable
        self.is_eagle = params.is_eagle
        self.kv_events = KVCacheEventRecorder(
            enabled=params.enable_kv_cache_events, page_size=self.page_size
        )

        if self.token_to_kv_pool_allocator:
            self.device = self.token_to_kv_pool_allocator.device
        else:
            self.device = torch.device("cpu")

View on GitHub (pinned to 0132848349)

Solutions

1. Reproduce with a minimal request sequence and report upstream (ping @hanming-lu as the message says) with the assert details
2. Update to a newer sglang release where the SWA radix accounting bug may be fixed
3. As a workaround disable the periodic sanity check in debug tooling (not a fix) and restart the server after any corruption

try:
    tree.sanity_check()
except Exception as e:
    if 'SWA Radix tree sanity check failed' in str(e):
        capture_request_trace(); report_upstream('@hanming-lu')
    raise

When it happens

Trigger: Calling sanity_check() on SWARadixCache after cache mutations when internal counters (evictable_size vs lru list evictable size, node counts) diverge — e.g. a bug in insert/evict for SWA vs full layers.

Common situations: Running SWA models (e.g. Gemma) with hierarchical cache or unusual cache_hit/evict sequences; regressions from changes to radix cache eviction logic.

Related errors

• evictable_size() is not implemented; use full_evictable_size
• protected_size() is not implemented; use full_protected_size
• kv-canary: lut_len must be positive when has_swa_lut is True
• kv-canary: lut_len must be 0 when has_swa_lut is False
• unexpected hidden shape {list(hidden.shape)}, expected {expe

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3169c929412eb97a. Report an issue: GitHub.