{"record":{"id":"97ab64df322b73a2","repo":"RyanCodrai/turbovec","slug":"duplicate-ids","errorCode":null,"errorMessage":"duplicate ids","messagePattern":"duplicate ids","errorType":"validation","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"turbovec/src/id_map.rs","lineNumber":912,"sourceCode":"    ///\n    /// Used by [`crate::convert`], which decodes a file into codes plus\n    /// ids and needs to re-emit it: the ids are validated for duplicates\n    /// exactly as a load does, since a table with a repeat cannot answer\n    /// `remove` or `contains` unambiguously.\n    pub(crate) fn from_index_and_ids(\n        inner: TurboQuantIndex,\n        slot_to_id: Vec<u64>,\n    ) -> std::io::Result<Self> {\n        if slot_to_id.len() != inner.len() {\n            return Err(std::io::Error::new(\n                std::io::ErrorKind::InvalidData,\n                format!(\"{} ids for {} rows\", slot_to_id.len(), inner.len()),\n            ));\n        }\n        let mut sorted = slot_to_id.clone();\n        sorted.sort_unstable();\n        if sorted.windows(2).any(|w| w[0] == w[1]) {\n            return Err(std::io::Error::new(\n                std::io::ErrorKind::InvalidData,\n                \"duplicate ids\",\n            ));\n        }\n        Ok(Self {\n            inner,\n            slot_to_id,\n            id_to_slot: std::sync::OnceLock::new(),\n            sorted_ids: std::sync::Mutex::new(sorted),\n            deferred_added: std::sync::Mutex::new(Default::default()),\n        })\n    }\n\n    /// Shared tail of the path and byte v7 loaders. `path` is `None` for\n    /// a byte image, which is not a sync destination.\n    fn from_v7_load(mut l: crate::io_v7::V7Load, path: Option<&Path>) -> std::io::Result<Self> {\n        let slot_to_id = std::mem::take(&mut l.ids);\n        let inner = TurboQuantIndex::from_v7(l, path)?;","sourceCodeStart":894,"sourceCodeEnd":930,"githubUrl":"https://github.com/RyanCodrai/turbovec/blob/ccab9f325e6ce2a270a87daf01ae4e443bcf2d49/turbovec/src/id_map.rs#L894-L930","documentation":"from_index_and_ids rejects id vectors containing duplicates: after sorting, any equal adjacent pair triggers an InvalidData io::Error with message \"duplicate ids\". Duplicate external ids would make remove/contains ambiguous, so the constructor refuses them.","triggerScenarios":"Calling from_index_and_ids where slot_to_id contains the same u64 id at two or more slots — detected by sorted.windows(2).any(|w| w[0] == w[1]).","commonSituations":"Id-generation code reusing ids after removals; appending documents whose ids were already ingested; a buggy dedup step upstream that assigned the same id to multiple chunks.","solutions":["Deduplicate or regenerate ids so every row has a unique external id before constructing the IdMap.","Use a stable unique key per document (e.g. hash of content + chunk index) instead of a reused counter.","Check duplicates first: sort a clone and compare adjacent elements to locate offending ids."],"exampleFix":"// before\nlet map = IdMap::from_index_and_ids(index, ids)?; // Err: duplicate ids\n// after\nlet mut sorted = ids.clone(); sorted.sort_unstable();\nassert!(sorted.windows(2).all(|w| w[0] != w[1]), \"duplicate ids\");\nlet map = IdMap::from_index_and_ids(index, ids)?;","handlingStrategy":"validation","validationCode":"let mut s = ids.clone(); s.sort_unstable();\nif s.windows(2).any(|w| w[0] == w[1]) {\n    return Err(\"duplicate ids before IdMap build\".into());\n}","typeGuard":"fn ids_unique(ids: &[u64]) -> bool {\n    let mut s = ids.to_vec(); s.sort_unstable(); s.windows(2).all(|w| w[0] != w[1])\n}","tryCatchPattern":"let map = IdMap::from_index_and_ids(index, ids)\n    .map_err(|e| { log::error!(\"{e}\"); BuildError::DuplicateIds })?;","preventionTips":["Use globally unique, monotonically assigned ids (never reuse after removal).","Deduplicate documents by content hash before assigning ids.","Check uniqueness with a HashSet when generating ids."],"tags":["rust","id-map","duplicate-keys","io-error","validation"],"backgroundTag":"duplicate-ids","analyzedSha":"ccab9f325e6ce2a270a87daf01ae4e443bcf2d49","analyzedAt":"2026-09-06T08:39:18.516Z","contentChangedAt":"2026-09-06T08:39:18.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}