{"record":{"id":"cfb11296e84d261d","repo":"tikv/tikv","slug":"ime-cross-check-fail-key-should-exist-write-cf-n","errorCode":null,"errorMessage":"ime cross check fail(key should exist): write cf not match;\n                        cache_region={:?}; disk_key={:?}, disk_mvcc={}; sequence_numer={}; prev_key_info={:?}","messagePattern":"ime cross check fail\\(key should exist\\): write cf not match;\n                        cache_region=(.+?); disk_key=(.+?), disk_mvcc=(.+?); sequence_numer=(.+?); prev_key_info=(.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"components/in_memory_engine/src/cross_check.rs","lineNumber":641,"sourceCode":"\n            let (disk_user_key, disk_mvcc) = split_ts(disk_iter.key()).unwrap();\n            // We cannot miss any types of write if the mvcc version is larger\n            // than `safe_point` of the relevant region. But the safe\n            // point can be updated during the cross check. Fetch it\n            // and check again.\n            if disk_mvcc > *safe_point {\n                *safe_point = {\n                    let region_maps = engine.core().region_manager().regions_map().read();\n                    let meta = region_maps.region_meta(cached_region.id).unwrap();\n                    // region might have split\n                    if meta.get_region() != cached_region {\n                        return Err(StopReason::RegionMetaChanged);\n                    }\n                    assert!(meta.safe_point() >= *safe_point);\n                    meta.safe_point()\n                };\n                if disk_mvcc > *safe_point {\n                    panic!(\n                        \"ime cross check fail(key should exist): write cf not match;\n                        cache_region={:?}; disk_key={:?}, disk_mvcc={}; sequence_numer={}; prev_key_info={:?}\",\n                        cached_region,\n                        log_wrappers::Value(disk_iter.key()),\n                        disk_mvcc,\n                        mem_iter.sequence_number,\n                        prev_key_info,\n                    );\n                }\n            }\n            let write = match parse_write(disk_iter.value()) {\n                Ok(write) => write,\n                Err(e) => {\n                    panic!(\n                        \"ime cross check fail(parse error);\n                        cache_region={:?}; cache_key={:?}, cache_val={:?}; sequence_numer={}; Error={:?}\",\n                        cached_region,\n                        log_wrappers::Value(mem_iter.key()),","sourceCodeStart":623,"sourceCodeEnd":659,"githubUrl":"https://github.com/tikv/tikv/blob/78aedc1c81ef3f7d8bacc6e9d09f56460f134937/components/in_memory_engine/src/cross_check.rs#L623-L659","documentation":"A panic in the in-memory engine's cross-check logic (cross_check.rs, check_remain_disk_key). While verifying that every valid (non-deleted, above-safe-point) write in the disk engine also exists in the cached region's memtable, the checker found a disk write record whose MVCC commit timestamp is above the region's safe point yet cannot be matched in the cache — i.e. a key that should be present in the in-memory cache is missing. This indicates the cache and disk engines have diverged for a loaded region, so the process panics rather than serve inconsistent data.","triggerScenarios":"Running the IME cross-check task over a region whose cache was loaded from disk; a write-CF record on disk with disk_mvcc > safe_point has no corresponding entry in the memtable iterator (mem_iter) for the cached region, so the assumed invariant 'every visible disk version is cached' is violated.","commonSituations":"Bugs or races in region load/evict paths (partial load, cache evicted mid-flight), write CF compaction or deletion racing with cache population, safe_point bookkeeping mistakes in the region manager, or data corruption/truncation of the in-memory write CF.","solutions":["Report the panic with the full cache_region, disk_key, disk_mvcc and prev_key_info output to TiKV maintainers; this is an internal invariant failure, not a user-recoverable error","Verify the region's load/evict lifecycle logs to see whether an evict or load raced with the cross-check","Check IME config (in-memory engine capacity, safe_point update interval) and consider disabling the in-memory engine to fall back to disk-only reads","Upgrade to a TiKV version with the relevant IME cross-check/evict race fixes"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Before enabling IME cross-check assumptions, verify region cache health:\nfn region_cache_consistent(disk_mvcc: u64, safe_point: u64, cached: bool) -> bool {\n    disk_mvcc <= safe_point || cached // every visible disk version must be cached\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Monitor IME cross-check logs for near-threshold safe_point/mvcc mismatches before they panic","Avoid concurrent evict/load operations on the same region while cross-check runs","Keep IME capacity sized so regions are not evicted mid-load","Pin to TiKV releases with known IME race fixes"],"tags":["tikv","in-memory-engine","cross-check","mvcc","panic"],"backgroundTag":"cache-disk-inconsistency","analyzedSha":"78aedc1c81ef3f7d8bacc6e9d09f56460f134937","analyzedAt":"2026-09-03T23:31:32.398Z","contentChangedAt":"2026-09-03T23:31:32.398Z","schemaVersion":2},"datasetVersion":"2026-09-11T07:07:21.782Z"}