{"record":{"id":"d30a17df10e7fa20","repo":"databendlabs/databend","slug":"duplicated-generickv-key-found-at-lines-and","errorCode":null,"errorMessage":"duplicated GenericKV key found at lines {} and {}: {}","messagePattern":"duplicated GenericKV key found at lines (.+?) and (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/process/src/filter_tenant.rs","lineNumber":258,"sourceCode":"                kind,\n                mark: None,\n            });\n            dump.report.state_machine_lines += 1;\n        }\n\n        Ok(dump)\n    }\n\n    fn build_indexes(&mut self) -> anyhow::Result<()> {\n        self.key_to_state.clear();\n        self.expires_by_key.clear();\n        self.index_ids_by_table_id.clear();\n\n        for (state_index, state_line) in self.state_lines.iter().enumerate() {\n            match &state_line.kind {\n                StateKind::GenericKV { key, .. } => {\n                    if let Some(prev) = self.key_to_state.insert(key.clone(), state_index) {\n                        anyhow::bail!(\n                            \"duplicated GenericKV key found at lines {} and {}: {}\",\n                            self.state_lines[prev].line_no,\n                            state_line.line_no,\n                            key\n                        );\n                    }\n                }\n                StateKind::Expire { key } => {\n                    self.expires_by_key\n                        .entry(key.clone())\n                        .or_default()\n                        .push(state_index);\n                }\n                StateKind::System { .. } => {}\n            }\n        }\n\n        for state_line in &self.state_lines {","sourceCodeStart":240,"sourceCodeEnd":276,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/meta/process/src/filter_tenant.rs#L240-L276","documentation":"While building lookup indexes over the parsed meta dump, `build_indexes` inserts every GenericKV key into `key_to_state`. If the same GenericKV key appears on two different lines, the second insert finds an existing entry and bails, reporting both line numbers and the key. The tool assumes the dump contains no duplicated GenericKV keys.","triggerScenarios":"Filtering a meta dump whose file contains the same GenericKV key twice — e.g. concatenating two snapshots, replaying lines twice, or a dump produced from an inconsistent state.","commonSituations":"Manually merged or appended dump files; re-running an export that appends instead of overwrites; snapshots taken across a Raft state-machine rebuild that legitimately contain duplicate keys.","solutions":["Deduplicate the dump file, keeping the last occurrence of each GenericKV key before running filter_tenant.","Check how the dump was produced/exported and regenerate it as a single consistent snapshot.","If duplicates are expected in your flow, relax build_indexes to keep the latest line instead of bailing."],"exampleFix":"// before\nif let Some(prev) = self.key_to_state.insert(key.clone(), state_index) {\n    anyhow::bail!(\"duplicated GenericKV key found at lines {} and {}: {}\", ...);\n}\n// after\n// keep the latest occurrence instead of failing\nself.key_to_state.insert(key.clone(), state_index);","handlingStrategy":"validation","validationCode":"# pre-flight: find duplicated GenericKV keys in the dump before filtering\ngrep -oE 'GenericKV \\{ key: [^,]+' dump.log | sort | uniq -d","typeGuard":null,"tryCatchPattern":"match tool.build_indexes() {\n    Err(e) if e.to_string().contains(\"duplicated GenericKV key\") => {\n        eprintln!(\"deduplicate the dump (keep last occurrence) and rerun\");\n        dedup_keep_last(\"dump.log\", \"dump.dedup.log\")?;\n        tool = Tool::load(\"dump.dedup.log\")?;\n        tool.build_indexes()?\n    }\n    other => other?,\n}","preventionTips":["Never concatenate multiple meta snapshots into one dump.","Overwrite (not append) when exporting state dumps.","Deduplicate keys before feeding the tool.","If your meta state legitimately duplicates keys, patch build_indexes to keep the latest line."],"tags":["meta","raft","deduplication","data-integrity"],"backgroundTag":"internal-invariant-violation","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}