tikv/tikv · critical

ime cross check fail(key should exist): write cf not match;

Error message

ime cross check fail(key should exist): write cf not match;
                        cache_region={:?}; disk_key={:?}, disk_mvcc={}; sequence_numer={}; prev_key_info={:?}

What it means

A panic in the in-memory engine's cross-check logic (cross_check.rs, check_remain_disk_key). While verifying that every valid (non-deleted, above-safe-point) write in the disk engine also exists in the cached region's memtable, the checker found a disk write record whose MVCC commit timestamp is above the region's safe point yet cannot be matched in the cache — i.e. a key that should be present in the in-memory cache is missing. This indicates the cache and disk engines have diverged for a loaded region, so the process panics rather than serve inconsistent data.

Source

Thrown at components/in_memory_engine/src/cross_check.rs:641

            let (disk_user_key, disk_mvcc) = split_ts(disk_iter.key()).unwrap();
            // We cannot miss any types of write if the mvcc version is larger
            // than `safe_point` of the relevant region. But the safe
            // point can be updated during the cross check. Fetch it
            // and check again.
            if disk_mvcc > *safe_point {
                *safe_point = {
                    let region_maps = engine.core().region_manager().regions_map().read();
                    let meta = region_maps.region_meta(cached_region.id).unwrap();
                    // region might have split
                    if meta.get_region() != cached_region {
                        return Err(StopReason::RegionMetaChanged);
                    }
                    assert!(meta.safe_point() >= *safe_point);
                    meta.safe_point()
                };
                if disk_mvcc > *safe_point {
                    panic!(
                        "ime cross check fail(key should exist): write cf not match;
                        cache_region={:?}; disk_key={:?}, disk_mvcc={}; sequence_numer={}; prev_key_info={:?}",
                        cached_region,
                        log_wrappers::Value(disk_iter.key()),
                        disk_mvcc,
                        mem_iter.sequence_number,
                        prev_key_info,
                    );
                }
            }
            let write = match parse_write(disk_iter.value()) {
                Ok(write) => write,
                Err(e) => {
                    panic!(
                        "ime cross check fail(parse error);
                        cache_region={:?}; cache_key={:?}, cache_val={:?}; sequence_numer={}; Error={:?}",
                        cached_region,
                        log_wrappers::Value(mem_iter.key()),

View on GitHub (pinned to 78aedc1c81)

Solutions

1. Report the panic with the full cache_region, disk_key, disk_mvcc and prev_key_info output to TiKV maintainers; this is an internal invariant failure, not a user-recoverable error
2. Verify the region's load/evict lifecycle logs to see whether an evict or load raced with the cross-check
3. Check IME config (in-memory engine capacity, safe_point update interval) and consider disabling the in-memory engine to fall back to disk-only reads
4. Upgrade to a TiKV version with the relevant IME cross-check/evict race fixes

// Before enabling IME cross-check assumptions, verify region cache health:
fn region_cache_consistent(disk_mvcc: u64, safe_point: u64, cached: bool) -> bool {
    disk_mvcc <= safe_point || cached // every visible disk version must be cached
}

When it happens

Trigger: Running the IME cross-check task over a region whose cache was loaded from disk; a write-CF record on disk with disk_mvcc > safe_point has no corresponding entry in the memtable iterator (mem_iter) for the cached region, so the assumed invariant 'every visible disk version is cached' is violated.

Common situations: Bugs or races in region load/evict paths (partial load, cache evicted mid-flight), write CF compaction or deletion racing with cache population, safe_point bookkeeping mistakes in the region manager, or data corruption/truncation of the in-memory write CF.

Related errors

• ime cross check fail(key should exist): miss valid mvcc vers
• ime cross check fail(key should exist): miss valid mvcc vers
• ime cross check fail(key should exist): miss valid mvcc vers
• ime cross check fail(key should exist): default not found;
• ime cross check fail(key should not exist): {:?} cf not matc

AI-assisted analysis of tikv/tikv@78aedc1c81 (2026-09-03). Data as JSON: /api/errors/cfb11296e84d261d. Report an issue: GitHub.