{"record":{"id":"bad67007d38008ca","repo":"databendlabs/databend","slug":"conflicting-tenant-filter-marks-for-line","errorCode":null,"errorMessage":"conflicting tenant filter marks for line {} ({}): existing {} because {}; new {} because {}","messagePattern":"conflicting tenant filter marks for line (.+?) \\((.+?)\\): existing (.+?) because (.+?); new (.+?) because (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/process/src/filter_tenant.rs","lineNumber":558,"sourceCode":"            self.mark_expires_for_key(key, decision, format!(\"expire of {key}\"))?;\n        }\n\n        Ok(marked)\n    }\n\n    fn mark_line_by_index(\n        &mut self,\n        index: usize,\n        decision: Decision,\n        reason: impl Into<String>,\n    ) -> anyhow::Result<bool> {\n        let reason = reason.into();\n        let state_line = &mut self.state_lines[index];\n\n        match &state_line.mark {\n            Some(mark) if mark.decision == decision => return Ok(false),\n            Some(mark) => {\n                anyhow::bail!(\n                    \"conflicting tenant filter marks for line {} ({}): existing {} because {}; new {} because {}\",\n                    state_line.line_no,\n                    state_line.display_key(),\n                    mark.decision.as_str(),\n                    mark.reason,\n                    decision.as_str(),\n                    reason\n                );\n            }\n            None => {}\n        }\n\n        state_line.mark = Some(Mark { decision, reason });\n\n        if let StateKind::GenericKV { key, .. } = &state_line.kind {\n            self.mark_queue.push_back((key.clone(), decision));\n        }\n","sourceCodeStart":540,"sourceCodeEnd":576,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/meta/process/src/filter_tenant.rs#L540-L576","documentation":"mark_line_by_index refuses to overwrite an existing mark that disagrees with the new decision. When a second pass (root classification, dependency marking, or expiry marking) tries to Drop a line already marked Keep — or vice versa — the tool's keep/drop logic is self-contradictory, so it bails with both the existing and new decision plus their reasons. This is a deliberate consistency check: a conflicting mark would mean the filtered dump could silently remove data another kept record depends on.","triggerScenarios":"Called from mark_all (system entries / root classification), mark_existing_key (via mark_required_key / mark_optional_key from drain_mark_queue), or mark_expires_for_key. The bail fires whenever state_lines[index].mark is Some with a different Decision than the incoming one, e.g. a dependency of a dropped key was already marked Keep by another root, or an __fd_*_expires key disagrees with its base key's decision.","commonSituations":"Snapshot dumps where cross-tenant or dangling references make one record both a required child of a kept key and a dependent of a dropped key; bugs in classify_root returning different decisions for keys that map to the same line; hand-edited dumps; changes to dependency_keys that create keep/drop cycles.","solutions":["Read the existing vs new decisions and reasons in the message to identify which two code paths disagree about the line.","Inspect the key's references in the dump: if it is genuinely shared, ensure dependency marking propagates the winning (Keep) decision first, or drop the whole dependent group together.","Fix classify_root/dependency logic so the decision for a key is deterministic and order-independent (e.g. Keep wins over Drop for shared required records).","Regenerate the dump with a matching metasrv version; version-skewed dumps often contain references that look contradictory to the filter.","As a diagnostic, log the full mark_queue order for the offending key before the bail to find the second caller."],"exampleFix":"// before: blindly marks, conflicting decisions abort\nself.mark_required_key(&child.key, decision, reason)?;\n// after: keep wins for shared required records\nlet existing = self.state_lines[self.key_to_state[&child.key]].mark.as_ref();\nif existing.map(|m| m.decision) == Some(Decision::Keep) && decision == Decision::Drop {\n    return Ok(false);\n}\nself.mark_required_key(&child.key, decision, reason)?;","handlingStrategy":"validation","validationCode":"fn decision_for_key(filter: &TenantFilter, key: &str) -> Option<Decision> {\n    filter.mark_of(key).map(|m| m.decision)\n}\n// call before re-marking:\nif decision_for_key(&filter, &key) == Some(Decision::Keep) {\n    eprintln!(\"key {key} already kept; skipping drop\");\n    return Ok(());\n}","typeGuard":"fn is_conflict(existing: &Option<Mark>, new: Decision) -> bool {\n    existing.as_ref().is_some_and(|m| m.decision != new)\n}","tryCatchPattern":"match filter.mark_line_by_index(idx, decision, reason) {\n    Ok(marked) => { if marked { propagate_to_dependencies(key); } }\n    Err(e) if e.to_string().contains(\"conflicting tenant filter marks\") => {\n        // decide deterministically: Keep wins for shared records\n        eprintln!(\"conflict on {key}; resolving as Keep\");\n        filter.mark_line_by_index(idx, Decision::Keep, \"conflict resolved: keep wins\")?;\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Make key decisions deterministic and order-independent: define explicitly whether Keep or Drop wins for shared records.","Mark Keep decisions before Drop passes so shared dependencies are protected first.","Include the original reason string in every mark to make conflicts diagnosable from the error alone.","Test dependency graphs containing shared records (one key referenced by both kept and dropped roots).","Never hand-edit dumps; conflicting references usually come from corrupted or version-skipped exports."],"tags":["rust","meta","conflicting-state","snapshot"],"backgroundTag":"invalid-state-transition","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}