{"record":{"id":"3169c929412eb97a","repo":"sgl-project/sglang","slug":"swa-radix-tree-sanity-check-failed-ping-hanming","errorCode":null,"errorMessage":"SWA Radix tree sanity check failed, ping @hanming-lu: {e}","messagePattern":"SWA Radix tree sanity check failed, ping @hanming-lu: (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/swa_radix_cache.py","lineNumber":342,"sourceCode":"                assert (\n                    x_lru.swa_lock_ref == 0\n                ), f\"x_lru should not be locked when idle, {x_lru.swa_lock_ref=}, {x_lru.swa_uuid=}, {x_lru.id=}\"\n                x_lru = getattr(x, self.prv)\n\n            if self.is_swa_list:\n                evictable_size = tree_cache.swa_evictable_size()\n                lru_list_evictable_size = self.sanity_check_evictable_size()\n            else:\n                evictable_size = tree_cache.full_evictable_size()\n                lru_list_evictable_size = self.sanity_check_evictable_size()\n\n            assert (\n                evictable_size == lru_list_evictable_size\n            ), f\"{self.is_swa_list=}, total nodes: {total_nodes}, total lru plus 1: {total_lru_plus_1}, evictable size: {evictable_size} != lru list evictable size: {lru_list_evictable_size}\"\n        except Exception as e:\n            msg = f\"SWA Radix tree sanity check failed, ping @hanming-lu: {e}\"\n            logger.error(msg)\n            raise Exception(msg)\n\n\nclass SWARadixCache(BasePrefixCache):\n    def __init__(self, params: CacheInitParams):\n        assert isinstance(params.token_to_kv_pool_allocator, SWATokenToKVPoolAllocator)\n        self.req_to_token_pool = params.req_to_token_pool\n        self.token_to_kv_pool_allocator = params.token_to_kv_pool_allocator\n        self.page_size = params.page_size\n        self.disable = params.disable\n        self.is_eagle = params.is_eagle\n        self.kv_events = KVCacheEventRecorder(\n            enabled=params.enable_kv_cache_events, page_size=self.page_size\n        )\n\n        if self.token_to_kv_pool_allocator:\n            self.device = self.token_to_kv_pool_allocator.device\n        else:\n            self.device = torch.device(\"cpu\")","sourceCodeStart":324,"sourceCodeEnd":360,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/swa_radix_cache.py#L324-L360","documentation":"The SWA radix cache's internal sanity check (a series of asserts comparing evictable sizes and LRU list sizes) failed. This indicates an internal consistency bug in tree/LRU bookkeeping; the message asks to ping @hanming-lu because it is not expected to be user-fixable via config.","triggerScenarios":"Calling sanity_check() on SWARadixCache after cache mutations when internal counters (evictable_size vs lru list evictable size, node counts) diverge — e.g. a bug in insert/evict for SWA vs full layers.","commonSituations":"Running SWA models (e.g. Gemma) with hierarchical cache or unusual cache_hit/evict sequences; regressions from changes to radix cache eviction logic.","solutions":["Reproduce with a minimal request sequence and report upstream (ping @hanming-lu as the message says) with the assert details","Update to a newer sglang release where the SWA radix accounting bug may be fixed","As a workaround disable the periodic sanity check in debug tooling (not a fix) and restart the server after any corruption"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    tree.sanity_check()\nexcept Exception as e:\n    if 'SWA Radix tree sanity check failed' in str(e):\n        capture_request_trace(); report_upstream('@hanming-lu')\n    raise","preventionTips":["Run sanity_check in staging workloads to catch divergence before production","Keep sglang updated for SWA radix fixes","Retain request logs to reproduce the mutating sequence that corrupts counters"],"tags":["swa","radix-cache","internal-bug","sanity-check"],"backgroundTag":"internal-invariant-violation","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}