{"record":{"id":"ef6bcfc183103b4f","repo":"databendlabs/databend","slug":"key-is-missing-from-state-index-key","errorCode":null,"errorMessage":"key is missing from state index: {key}","messagePattern":"key is missing from state index: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/process/src/filter_tenant.rs","lineNumber":318,"sourceCode":"        for index in 0..self.state_lines.len() {\n            if matches!(self.state_lines[index].kind, StateKind::System { .. }) {\n                self.mark_line_by_index(index, Decision::Keep, \"system state machine entry\")?;\n            }\n        }\n\n        let keys = self.key_to_state.keys().cloned().collect::<Vec<_>>();\n        for key in keys {\n            if let Some(root) = classify_root(&key, tenant) {\n                self.mark_required_key(&key, root.decision, root.reason)?;\n            }\n        }\n\n        self.drain_mark_queue()?;\n\n        let keys = self.key_to_state.keys().cloned().collect::<Vec<_>>();\n        for key in keys {\n            let Some(index) = self.key_to_state.get(&key).copied() else {\n                anyhow::bail!(\"key is missing from state index: {key}\");\n            };\n            if self.state_lines[index].mark.is_some() {\n                continue;\n            }\n\n            if let Some(reason) = self.classify_snapshot_orphan_root(&key)? {\n                eprintln!(\n                    \"filter-tenant: drop snapshot orphan at line {}: {}: {}\",\n                    self.state_lines[index].line_no, key, reason\n                );\n                self.mark_required_key(&key, Decision::Drop, reason)?;\n            }\n        }\n\n        self.drain_mark_queue()?;\n        self.assert_every_state_line_marked()?;\n        self.update_report();\n","sourceCodeStart":300,"sourceCodeEnd":336,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/meta/process/src/filter_tenant.rs#L300-L336","documentation":"`mark_all` iterates all keys recorded in `key_to_state` and looks each up again to find its state line index. If a key present in the key map cannot be resolved to a state line index (map/queue inconsistency, e.g. indexes invalidated while a mark queue was draining), it bails with this message naming the key. This is an internal consistency check of the filter's bookkeeping.","triggerScenarios":"An internal invariant break where key_to_state and state_lines get out of sync — most plausibly caused by concurrent mutation or by drain_mark_queue/classify_snapshot_orphan_root mutating the index while mark_all is iterating.","commonSituations":"Modifying filter_tenant to mutate key_to_state or state_lines (e.g. removing lines) without updating the mark queue logic; custom patches to the tool.","solutions":["Use an unmodified filter_tenant build; this should be unreachable on stock code paths.","If you patched the code, ensure any mutation of key_to_state/state_lines happens only after drain_mark_queue and before collecting keys, or re-collect keys after mutations.","Debug by dumping the key and the length of state_lines to find where the index desync originates."],"exampleFix":"// before\nlet Some(index) = self.key_to_state.get(&key).copied() else {\n    anyhow::bail!(\"key is missing from state index: {key}\");\n};\n// after\nlet Some(index) = self.key_to_state.get(&key).copied() else {\n    tracing::warn!(\"skipping stale key in state index: {key}\");\n    continue;\n};","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"match tool.mark_all(reason) {\n    Err(e) if e.to_string().contains(\"key is missing from state index\") => {\n        eprintln!(\"internal index desync: {}\", e);\n        std::process::exit(4); // rerun from a fresh load\n    }\n    other => other?,\n}","preventionTips":["Do not mutate key_to_state/state_lines while mark_all is running.","Call drain_mark_queue only at a point where the index is stable.","After any code change that removes state lines, rebuild key_to_state before marking.","Treat this message as a bug report trigger for the tool maintainers with the offending key attached."],"tags":["meta","raft","invariant","bookkeeping"],"backgroundTag":"internal-invariant-violation","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}