{"record":{"id":"d3598d1212adee83","repo":"databendlabs/databend","slug":"marked-key-is-missing-from-state-index-key","errorCode":null,"errorMessage":"marked key is missing from state index: {key}","messagePattern":"marked key is missing from state index: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/process/src/filter_tenant.rs","lineNumber":600,"sourceCode":"        key: &str,\n        decision: Decision,\n        reason: impl Into<String>,\n    ) -> anyhow::Result<()> {\n        let Some(indices) = self.expires_by_key.get(key).cloned() else {\n            return Ok(());\n        };\n\n        let reason = reason.into();\n        for index in indices {\n            self.mark_line_by_index(index, decision, reason.clone())?;\n        }\n\n        Ok(())\n    }\n\n    fn dependency_keys(&self, key: &str) -> anyhow::Result<Vec<DependencyKey>> {\n        let Some(index) = self.key_to_state.get(key).copied() else {\n            anyhow::bail!(\"marked key is missing from state index: {key}\");\n        };\n\n        let StateKind::GenericKV { value, .. } = &self.state_lines[index].kind else {\n            anyhow::bail!(\n                \"marked key is not a GenericKV state entry at line {}: {}\",\n                self.state_lines[index].line_no,\n                self.state_lines[index].display_key()\n            );\n        };\n        let data = &value.data;\n\n        let mut out = DependencyKeySet::default();\n        let segments = split_key(key);\n\n        match segments.as_slice() {\n            [\"__fd_database\", _tenant, _db_name] => {\n                let db_id = decode_json_u64(key, data)?;\n                self.add_database_id_keys(db_id, &mut out);","sourceCodeStart":582,"sourceCodeEnd":618,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/meta/process/src/filter_tenant.rs#L582-L618","documentation":"This error is raised by `dependency_keys` in the tenant-filtering meta migration tool when a key that was queued on the mark queue cannot be found in the in-memory `key_to_state` index. The index maps every parsed state line's key to its position in `state_lines`, so a mark-queue entry without a corresponding index entry means the internal bookkeeping is inconsistent. It is an internal invariant check, not a user-facing validation error.","triggerScenarios":"Calling `drain_mark_queue` while it processes a marked key that was never inserted into `key_to_state` (or was inserted for a different key string), e.g. due to a bug in the mark/collect pass or a duplicate/renamed key during state parsing.","commonSituations":"Running the filter-tenant migration over a meta snapshot whose parsing pass skipped or deduplicated a line that the mark queue still references; hand-edited or partially regenerated snapshot files; code changes that add keys to the mark queue without registering them in `key_to_state`.","solutions":["Inspect the reported key and check how it was added to the mark queue; ensure every enqueued key is also inserted into `key_to_state` during state parsing.","Re-run the migration on a freshly parsed snapshot to rule out stale/partial in-memory state.","Check for key normalization mismatches (case, quoting, whitespace) between the mark queue and the state index.","If triggered by a code change, fix the producer so mark-queue entries and index entries are always inserted together.","Report upstream with the snapshot and key if the invariant fails on unmodified tooling."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Rust: verify the key is indexed before draining the mark queue\nfn ensure_indexed(key_to_state: &HashMap<String, usize>, key: &str) -> Result<(), String> {\n    if key_to_state.contains_key(key) { Ok(()) } else { Err(format!(\"key not indexed: {key}\")) }\n}","typeGuard":"fn is_indexed(key_to_state: &HashMap<String, usize>, key: &str) -> bool {\n    key_to_state.contains_key(key)\n}","tryCatchPattern":"match tool.drain_mark_queue() {\n    Ok(deps) => deps,\n    Err(e) if e.to_string().contains(\"marked key is missing from state index\") => {\n        eprintln!(\"internal state index inconsistency: {e}; re-parse snapshot\");\n        return Err(e.into());\n    }\n    Err(e) => return Err(e.into()),\n}","preventionTips":["Keep mark-queue insertion and key_to_state insertion adjacent in the same code path","Add a debug assertion or self-check that every queued key is indexed before draining","Avoid hand-editing parsed snapshot state","Pin the tool version to the meta data format version"],"tags":["internal-invariant","meta","state-index","migration"],"backgroundTag":"internal-invariant-violation","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}