tenant dump filter did not mark all state machine entries…

Error message

tenant dump filter did not mark all state machine entries; examples: {}

What it means

filter_tenant's contract is that after mark_all finishes, every state-machine line in the dump has been explicitly marked Keep or Drop. assert_every_state_line_marked is the final safety gate before writing the filtered dump: if any line was left unmarked, the tool would silently drop or mishandle it, so it bails listing up to 20 example unmarked lines (line number + key).

Solutions

Read the example line numbers/keys in the message, locate them in the dump file, and identify the unrecognized key shape.
Add the missing key pattern to classify_root (with a Keep/Drop decision) or to classify_snapshot_orphan_root / dependency_keys so the line gets marked.
Regenerate the dump with a metasrv version matching the filter_tenant tool to rule out format drift.
As a stopgap, check whether the unmarked lines belong to the tenant being dropped and can be safely excluded upstream; do not hand-edit marks in the dump file.

Example fix

// before: unknown key shape falls through
_ => None,
// after: explicitly classify the new prefix in classify_root
["__fd_new_record", _id] => Some(Decision::Keep, "known new record"),
_ => None,

Defensive patterns

Strategy: validation

Validate before calling

// before running the filter, check for unknown key prefixes yourself
let known = ["__fd_table", "__fd_index_by_id", "__fd_marked_deleted_index",
             "__fd_marked_deleted_table_index", "__fd_table_copied_file_lock",
             "__fd_table_lvt", "__fd_table_copied_files", "__fd_table_tag"];
let unknown: Vec<&str> = keys
    .iter()
    .filter(|k| !k.starts_with("$") && !known.iter().any(|p| k.starts_with(p)))
    .map(|k| k.as_str())
    .collect();
anyhow::ensure!(unknown.is_empty(), "unknown key shapes: {:?}", unknown);

Type guard

fn is_classifiable(key: &str, known_prefixes: &[&str]) -> bool {
    key.starts_with('$') || known_prefixes.iter().any(|p| key.starts_with(p))
}

Try / catch

match filter.run(tenant) {
    Ok(report) => println!("kept {}, dropped {}", report.kept, report.dropped),
    Err(e) if e.to_string().contains("did not mark all state machine entries") => {
        // dump contains key shapes this tool version does not know; report and stop
        eprintln!("incompatible dump: {e}");
    }
    Err(e) => return Err(e),
}

Prevention

Validate dump key shapes against the tool's supported prefix list before filtering.
When metasrv adds new state record types, extend classify_root/dependency_keys in the same change.
Run the filter tool inside CI against a fixture snapshot so unclassified keys surface immediately.
Check the full error message for all example lines, not just the first, to identify every missing pattern at once.

When it happens

Trigger: filter_tenant::mark_all -> assert_every_state_line_marked finds lines with mark == None after all classification passes: system entries, classify_root results, drain_mark_queue dependency marking, and classify_snapshot_orphan_root orphans. Any key prefix not covered by classify_root, dependency_keys, or the orphan classifier leaves its line unmarked.

Common situations: Running the tool against a snapshot dump containing KV key shapes from a newer metasrv version (new __fd_* prefixes or state records the classifier does not know); corrupted or truncated dump lines; a code change that removed a classification branch.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/691538556d73b57f. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/process/src/filter_tenant.rs:495

            .filter(|line| {
                line.mark
                    .as_ref()
                    .is_some_and(|mark| mark.decision == Decision::Drop)
            })
            .count();
    }

    fn assert_every_state_line_marked(&self) -> anyhow::Result<()> {
        let unmarked = self
            .state_lines
            .iter()
            .filter(|line| line.mark.is_none())
            .take(20)
            .map(|line| format!("line {}: {}", line.line_no, line.display_key()))
            .collect::<Vec<_>>();

        if !unmarked.is_empty() {
            anyhow::bail!(
                "tenant dump filter did not mark all state machine entries; examples: {}",
                unmarked.join("; ")
            );
        }

        Ok(())
    }

    fn mark_required_key(
        &mut self,
        key: &str,
        decision: Decision,
        reason: impl Into<String>,
    ) -> anyhow::Result<bool> {
        let reason = reason.into();
        let Some(index) = self.key_to_state.get(key).copied() else {
            anyhow::bail!("required key not found: {key}; reason: {reason}");
        };

View on GitHub (pinned to 288d84d76e)