{"record":{"id":"17303c5335ca1114","repo":"tikv/tikv","slug":"ime-cross-check-fail-key-should-exist-miss-valid-17303c","errorCode":null,"errorMessage":"ime cross check fail(key should exist): miss valid mvcc version;\n                            cache_region={:?}; disk_key={:?}; sequence_numer={}; read_ts={}, safe_point={}; prev_key_info={:?}","messagePattern":"ime cross check fail\\(key should exist\\): miss valid mvcc version;\n                            cache_region=(.+?); disk_key=(.+?); sequence_numer=(.+?); read_ts=(.+?), safe_point=(.+?); prev_key_info=(.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"components/in_memory_engine/src/cross_check.rs","lineNumber":749,"sourceCode":"                    // region might have split\n                    if meta.get_region() != cached_region {\n                        return Err(StopReason::RegionMetaChanged);\n                    }\n                    assert!(meta.safe_point() >= *safe_point);\n                    meta.safe_point()\n                };\n                prev_key_info.update_last_mvcc_version_before_safe_point(*safe_point);\n            }\n            if prev_key_info.last_mvcc_version_before_safe_point == 0 {\n                if disk_user_key != last_disk_user_key {\n                    *last_disk_user_key = disk_user_key.to_vec();\n                    *last_disk_user_key_delete = false;\n                }\n                if !*last_disk_user_key_delete {\n                    if write.write_type == WriteType::Delete {\n                        *last_disk_user_key_delete = true;\n                    } else {\n                        panic!(\n                            \"ime cross check fail(key should exist): miss valid mvcc version;\n                            cache_region={:?}; disk_key={:?}; sequence_numer={}; read_ts={}, safe_point={}; prev_key_info={:?}\",\n                            cached_region,\n                            log_wrappers::Value(disk_key),\n                            mem_iter.sequence_number,\n                            mem_iter.snapshot_read_ts,\n                            safe_point,\n                            prev_key_info,\n                        );\n                    }\n                }\n            } else {\n                if disk_mvcc > prev_key_info.last_mvcc_version_before_safe_point {\n                    if write.write_type == WriteType::Rollback\n                        || write.write_type == WriteType::Lock\n                    {\n                        info!(\n                            \"ime meet gced rollback or lock\";","sourceCodeStart":731,"sourceCodeEnd":767,"githubUrl":"https://github.com/tikv/tikv/blob/78aedc1c81ef3f7d8bacc6e9d09f56460f134937/components/in_memory_engine/src/cross_check.rs#L731-L767","documentation":"A panic in the IME cross-check (cross_check.rs, check_duplicated_mvcc_version_for_last_user_key). After seeing a valid (non-delete) write for the last user key at read_ts above safe_point, the checker found the newest version is a Delete — meaning a version that should be readable from the cache is missing a valid MVCC version, so the cached region lost data relative to the disk snapshot. The process panics because subsequent reads could return wrong results (missing key instead of an existing value).","triggerScenarios":"While walking duplicate MVCC versions of a user key: the previous disk version was a valid put, and the current write (from the cache iterator, mem_iter.sequence_number with snapshot_read_ts and safe_point in the message) turns out to be WriteType::Delete, breaking the invariant that visible versions above safe_point are preserved in cache.","commonSituations":"Eviction/load races dropping newer versions, MVCC GC or delete markers applied to the cache but not accounted for by safe_point, or corruption of cached write CF entries.","solutions":["File a TiKV issue with the panic output including prev_key_info; this is an internal invariant violation","Check for recent region evict/load events in the TiKV log around the panic time","Restart the node to reload the region cache from disk","Disable the in-memory engine or upgrade to a fixed release"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"fn visible_version_expected(write_type: WriteType, last_was_valid_put: bool) -> bool {\n    // a valid put must not be followed by an unexpected delete above safe_point\n    !(last_was_valid_put && write_type == WriteType::Delete)\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep safe_point updates in sync with cache-side deletes/GC","Watch region evict/load event logs for races preceding the check","Restart or disable IME to recover rather than retrying reads after the panic","Report with prev_key_info output included"],"tags":["tikv","in-memory-engine","cross-check","mvcc","panic"],"backgroundTag":"cache-disk-inconsistency","analyzedSha":"78aedc1c81ef3f7d8bacc6e9d09f56460f134937","analyzedAt":"2026-09-03T23:31:32.398Z","contentChangedAt":"2026-09-03T23:31:32.398Z","schemaVersion":2},"datasetVersion":"2026-09-11T07:07:21.782Z"}