{"record":{"id":"691538556d73b57f","repo":"databendlabs/databend","slug":"tenant-dump-filter-did-not-mark-all-state-machine","errorCode":null,"errorMessage":"tenant dump filter did not mark all state machine entries; examples: {}","messagePattern":"tenant dump filter did not mark all state machine entries; examples: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/process/src/filter_tenant.rs","lineNumber":495,"sourceCode":"            .filter(|line| {\n                line.mark\n                    .as_ref()\n                    .is_some_and(|mark| mark.decision == Decision::Drop)\n            })\n            .count();\n    }\n\n    fn assert_every_state_line_marked(&self) -> anyhow::Result<()> {\n        let unmarked = self\n            .state_lines\n            .iter()\n            .filter(|line| line.mark.is_none())\n            .take(20)\n            .map(|line| format!(\"line {}: {}\", line.line_no, line.display_key()))\n            .collect::<Vec<_>>();\n\n        if !unmarked.is_empty() {\n            anyhow::bail!(\n                \"tenant dump filter did not mark all state machine entries; examples: {}\",\n                unmarked.join(\"; \")\n            );\n        }\n\n        Ok(())\n    }\n\n    fn mark_required_key(\n        &mut self,\n        key: &str,\n        decision: Decision,\n        reason: impl Into<String>,\n    ) -> anyhow::Result<bool> {\n        let reason = reason.into();\n        let Some(index) = self.key_to_state.get(key).copied() else {\n            anyhow::bail!(\"required key not found: {key}; reason: {reason}\");\n        };","sourceCodeStart":477,"sourceCodeEnd":513,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/meta/process/src/filter_tenant.rs#L477-L513","documentation":"filter_tenant's contract is that after mark_all finishes, every state-machine line in the dump has been explicitly marked Keep or Drop. assert_every_state_line_marked is the final safety gate before writing the filtered dump: if any line was left unmarked, the tool would silently drop or mishandle it, so it bails listing up to 20 example unmarked lines (line number + key).","triggerScenarios":"filter_tenant::mark_all -> assert_every_state_line_marked finds lines with mark == None after all classification passes: system entries, classify_root results, drain_mark_queue dependency marking, and classify_snapshot_orphan_root orphans. Any key prefix not covered by classify_root, dependency_keys, or the orphan classifier leaves its line unmarked.","commonSituations":"Running the tool against a snapshot dump containing KV key shapes from a newer metasrv version (new __fd_* prefixes or state records the classifier does not know); corrupted or truncated dump lines; a code change that removed a classification branch.","solutions":["Read the example line numbers/keys in the message, locate them in the dump file, and identify the unrecognized key shape.","Add the missing key pattern to classify_root (with a Keep/Drop decision) or to classify_snapshot_orphan_root / dependency_keys so the line gets marked.","Regenerate the dump with a metasrv version matching the filter_tenant tool to rule out format drift.","As a stopgap, check whether the unmarked lines belong to the tenant being dropped and can be safely excluded upstream; do not hand-edit marks in the dump file."],"exampleFix":"// before: unknown key shape falls through\n_ => None,\n// after: explicitly classify the new prefix in classify_root\n[\"__fd_new_record\", _id] => Some(Decision::Keep, \"known new record\"),\n_ => None,","handlingStrategy":"validation","validationCode":"// before running the filter, check for unknown key prefixes yourself\nlet known = [\"__fd_table\", \"__fd_index_by_id\", \"__fd_marked_deleted_index\",\n             \"__fd_marked_deleted_table_index\", \"__fd_table_copied_file_lock\",\n             \"__fd_table_lvt\", \"__fd_table_copied_files\", \"__fd_table_tag\"];\nlet unknown: Vec<&str> = keys\n    .iter()\n    .filter(|k| !k.starts_with(\"$\") && !known.iter().any(|p| k.starts_with(p)))\n    .map(|k| k.as_str())\n    .collect();\nanyhow::ensure!(unknown.is_empty(), \"unknown key shapes: {:?}\", unknown);","typeGuard":"fn is_classifiable(key: &str, known_prefixes: &[&str]) -> bool {\n    key.starts_with('$') || known_prefixes.iter().any(|p| key.starts_with(p))\n}","tryCatchPattern":"match filter.run(tenant) {\n    Ok(report) => println!(\"kept {}, dropped {}\", report.kept, report.dropped),\n    Err(e) if e.to_string().contains(\"did not mark all state machine entries\") => {\n        // dump contains key shapes this tool version does not know; report and stop\n        eprintln!(\"incompatible dump: {e}\");\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Validate dump key shapes against the tool's supported prefix list before filtering.","When metasrv adds new state record types, extend classify_root/dependency_keys in the same change.","Run the filter tool inside CI against a fixture snapshot so unclassified keys surface immediately.","Check the full error message for all example lines, not just the first, to identify every missing pattern at once."],"tags":["rust","meta","unclassified-key","snapshot"],"backgroundTag":"internal-invariant-violation","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}