What causes "marked key is missing from state index: {key}" in databendlabs/databend?

Inspect the reported key and check how it was added to the mark queue; ensure every enqueued key is also inserted into `key_to_state` during state parsing.

databendlabs/databend · error

marked key is missing from state index

Error message

marked key is missing from state index: {key}

What it means

This error is raised by `dependency_keys` in the tenant-filtering meta migration tool when a key that was queued on the mark queue cannot be found in the in-memory `key_to_state` index. The index maps every parsed state line's key to its position in `state_lines`, so a mark-queue entry without a corresponding index entry means the internal bookkeeping is inconsistent. It is an internal invariant check, not a user-facing validation error.

Solutions

Inspect the reported key and check how it was added to the mark queue; ensure every enqueued key is also inserted into `key_to_state` during state parsing.
Re-run the migration on a freshly parsed snapshot to rule out stale/partial in-memory state.
Check for key normalization mismatches (case, quoting, whitespace) between the mark queue and the state index.
If triggered by a code change, fix the producer so mark-queue entries and index entries are always inserted together.
Report upstream with the snapshot and key if the invariant fails on unmodified tooling.

Defensive patterns

Strategy: try-catch

Validate before calling

// Rust: verify the key is indexed before draining the mark queue
fn ensure_indexed(key_to_state: &HashMap<String, usize>, key: &str) -> Result<(), String> {
    if key_to_state.contains_key(key) { Ok(()) } else { Err(format!("key not indexed: {key}")) }
}

Type guard

fn is_indexed(key_to_state: &HashMap<String, usize>, key: &str) -> bool {
    key_to_state.contains_key(key)
}

Try / catch

match tool.drain_mark_queue() {
    Ok(deps) => deps,
    Err(e) if e.to_string().contains("marked key is missing from state index") => {
        eprintln!("internal state index inconsistency: {e}; re-parse snapshot");
        return Err(e.into());
    }
    Err(e) => return Err(e.into()),
}

Prevention

Keep mark-queue insertion and key_to_state insertion adjacent in the same code path
Add a debug assertion or self-check that every queued key is indexed before draining
Avoid hand-editing parsed snapshot state
Pin the tool version to the meta data format version

When it happens

Trigger: Calling `drain_mark_queue` while it processes a marked key that was never inserted into `key_to_state` (or was inserted for a different key string), e.g. due to a bug in the mark/collect pass or a duplicate/renamed key during state parsing.

Common situations: Running the filter-tenant migration over a meta snapshot whose parsing pass skipped or deduplicated a line that the mark queue still references; hand-edited or partially regenerated snapshot files; code changes that add keys to the mark queue without registering them in `key_to_state`.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/d3598d1212adee83. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/process/src/filter_tenant.rs:600

        key: &str,
        decision: Decision,
        reason: impl Into<String>,
    ) -> anyhow::Result<()> {
        let Some(indices) = self.expires_by_key.get(key).cloned() else {
            return Ok(());
        };

        let reason = reason.into();
        for index in indices {
            self.mark_line_by_index(index, decision, reason.clone())?;
        }

        Ok(())
    }

    fn dependency_keys(&self, key: &str) -> anyhow::Result<Vec<DependencyKey>> {
        let Some(index) = self.key_to_state.get(key).copied() else {
            anyhow::bail!("marked key is missing from state index: {key}");
        };

        let StateKind::GenericKV { value, .. } = &self.state_lines[index].kind else {
            anyhow::bail!(
                "marked key is not a GenericKV state entry at line {}: {}",
                self.state_lines[index].line_no,
                self.state_lines[index].display_key()
            );
        };
        let data = &value.data;

        let mut out = DependencyKeySet::default();
        let segments = split_key(key);

        match segments.as_slice() {
            ["__fd_database", _tenant, _db_name] => {
                let db_id = decode_json_u64(key, data)?;
                self.add_database_id_keys(db_id, &mut out);

View on GitHub (pinned to 288d84d76e)