tikv/tikv · critical

ime cross check fail(key should exist): miss valid mvcc vers

Error message

ime cross check fail(key should exist): miss valid mvcc version;
                            cache_region={:?}; disk_key={:?}; sequence_numer={}; read_ts={}, safe_point={}; prev_key_info={:?}

What it means

A panic in the IME cross-check (cross_check.rs, check_duplicated_mvcc_version_for_last_user_key). After seeing a valid (non-delete) write for the last user key at read_ts above safe_point, the checker found the newest version is a Delete — meaning a version that should be readable from the cache is missing a valid MVCC version, so the cached region lost data relative to the disk snapshot. The process panics because subsequent reads could return wrong results (missing key instead of an existing value).

Source

Thrown at components/in_memory_engine/src/cross_check.rs:749

                    // region might have split
                    if meta.get_region() != cached_region {
                        return Err(StopReason::RegionMetaChanged);
                    }
                    assert!(meta.safe_point() >= *safe_point);
                    meta.safe_point()
                };
                prev_key_info.update_last_mvcc_version_before_safe_point(*safe_point);
            }
            if prev_key_info.last_mvcc_version_before_safe_point == 0 {
                if disk_user_key != last_disk_user_key {
                    *last_disk_user_key = disk_user_key.to_vec();
                    *last_disk_user_key_delete = false;
                }
                if !*last_disk_user_key_delete {
                    if write.write_type == WriteType::Delete {
                        *last_disk_user_key_delete = true;
                    } else {
                        panic!(
                            "ime cross check fail(key should exist): miss valid mvcc version;
                            cache_region={:?}; disk_key={:?}; sequence_numer={}; read_ts={}, safe_point={}; prev_key_info={:?}",
                            cached_region,
                            log_wrappers::Value(disk_key),
                            mem_iter.sequence_number,
                            mem_iter.snapshot_read_ts,
                            safe_point,
                            prev_key_info,
                        );
                    }
                }
            } else {
                if disk_mvcc > prev_key_info.last_mvcc_version_before_safe_point {
                    if write.write_type == WriteType::Rollback
                        || write.write_type == WriteType::Lock
                    {
                        info!(
                            "ime meet gced rollback or lock";

View on GitHub (pinned to 78aedc1c81)

Solutions

1. File a TiKV issue with the panic output including prev_key_info; this is an internal invariant violation
2. Check for recent region evict/load events in the TiKV log around the panic time
3. Restart the node to reload the region cache from disk
4. Disable the in-memory engine or upgrade to a fixed release

fn visible_version_expected(write_type: WriteType, last_was_valid_put: bool) -> bool {
    // a valid put must not be followed by an unexpected delete above safe_point
    !(last_was_valid_put && write_type == WriteType::Delete)
}

When it happens

Trigger: While walking duplicate MVCC versions of a user key: the previous disk version was a valid put, and the current write (from the cache iterator, mem_iter.sequence_number with snapshot_read_ts and safe_point in the message) turns out to be WriteType::Delete, breaking the invariant that visible versions above safe_point are preserved in cache.

Common situations: Eviction/load races dropping newer versions, MVCC GC or delete markers applied to the cache but not accounted for by safe_point, or corruption of cached write CF entries.

Related errors

• ime cross check fail(key should exist): write cf not match;
• ime cross check fail(key should exist): miss valid mvcc vers
• ime cross check fail(key should exist): miss valid mvcc vers
• ime cross check fail(key should exist): default not found;
• ime cross check fail(key should not exist): {:?} cf not matc

AI-assisted analysis of tikv/tikv@78aedc1c81 (2026-09-03). Data as JSON: /api/errors/17303c5335ca1114. Report an issue: GitHub.