databendlabs/databend · error

key is missing from state index

Error message

key is missing from state index: {key}

What it means

`mark_all` iterates all keys recorded in `key_to_state` and looks each up again to find its state line index. If a key present in the key map cannot be resolved to a state line index (map/queue inconsistency, e.g. indexes invalidated while a mark queue was draining), it bails with this message naming the key. This is an internal consistency check of the filter's bookkeeping.

Solutions

1. Use an unmodified filter_tenant build; this should be unreachable on stock code paths.
2. If you patched the code, ensure any mutation of key_to_state/state_lines happens only after drain_mark_queue and before collecting keys, or re-collect keys after mutations.
3. Debug by dumping the key and the length of state_lines to find where the index desync originates.

Example fix

// before
let Some(index) = self.key_to_state.get(&key).copied() else {
    anyhow::bail!("key is missing from state index: {key}");
};
// after
let Some(index) = self.key_to_state.get(&key).copied() else {
    tracing::warn!("skipping stale key in state index: {key}");
    continue;
};

match tool.mark_all(reason) {
    Err(e) if e.to_string().contains("key is missing from state index") => {
        eprintln!("internal index desync: {}", e);
        std::process::exit(4); // rerun from a fresh load
    }
    other => other?,
}

When it happens

Trigger: An internal invariant break where key_to_state and state_lines get out of sync — most plausibly caused by concurrent mutation or by drain_mark_queue/classify_snapshot_orphan_root mutating the index while mark_all is iterating.

Common situations: Modifying filter_tenant to mutate key_to_state or state_lines (e.g. removing lines) without updating the mark queue logic; custom patches to the tool.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• unsupported state machine entry at line
• duplicated GenericKV key found at lines
• stage_prefix should never be called on external stage, must…
• The header tree can only contain DataHeader
• _ => unreachable!()

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/ef6bcfc183103b4f. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/process/src/filter_tenant.rs:318

        for index in 0..self.state_lines.len() {
            if matches!(self.state_lines[index].kind, StateKind::System { .. }) {
                self.mark_line_by_index(index, Decision::Keep, "system state machine entry")?;
            }
        }

        let keys = self.key_to_state.keys().cloned().collect::<Vec<_>>();
        for key in keys {
            if let Some(root) = classify_root(&key, tenant) {
                self.mark_required_key(&key, root.decision, root.reason)?;
            }
        }

        self.drain_mark_queue()?;

        let keys = self.key_to_state.keys().cloned().collect::<Vec<_>>();
        for key in keys {
            let Some(index) = self.key_to_state.get(&key).copied() else {
                anyhow::bail!("key is missing from state index: {key}");
            };
            if self.state_lines[index].mark.is_some() {
                continue;
            }

            if let Some(reason) = self.classify_snapshot_orphan_root(&key)? {
                eprintln!(
                    "filter-tenant: drop snapshot orphan at line {}: {}: {}",
                    self.state_lines[index].line_no, key, reason
                );
                self.mark_required_key(&key, Decision::Drop, reason)?;
            }
        }

        self.drain_mark_queue()?;
        self.assert_every_state_line_marked()?;
        self.update_report();

View on GitHub (pinned to 288d84d76e)