RyanCodrai/turbovec · error · io::Error

duplicate ids

Error message

duplicate ids

What it means

from_index_and_ids rejects id vectors containing duplicates: after sorting, any equal adjacent pair triggers an InvalidData io::Error with message "duplicate ids". Duplicate external ids would make remove/contains ambiguous, so the constructor refuses them.

Source

Thrown at turbovec/src/id_map.rs:912

    ///
    /// Used by [`crate::convert`], which decodes a file into codes plus
    /// ids and needs to re-emit it: the ids are validated for duplicates
    /// exactly as a load does, since a table with a repeat cannot answer
    /// `remove` or `contains` unambiguously.
    pub(crate) fn from_index_and_ids(
        inner: TurboQuantIndex,
        slot_to_id: Vec<u64>,
    ) -> std::io::Result<Self> {
        if slot_to_id.len() != inner.len() {
            return Err(std::io::Error::new(
                std::io::ErrorKind::InvalidData,
                format!("{} ids for {} rows", slot_to_id.len(), inner.len()),
            ));
        }
        let mut sorted = slot_to_id.clone();
        sorted.sort_unstable();
        if sorted.windows(2).any(|w| w[0] == w[1]) {
            return Err(std::io::Error::new(
                std::io::ErrorKind::InvalidData,
                "duplicate ids",
            ));
        }
        Ok(Self {
            inner,
            slot_to_id,
            id_to_slot: std::sync::OnceLock::new(),
            sorted_ids: std::sync::Mutex::new(sorted),
            deferred_added: std::sync::Mutex::new(Default::default()),
        })
    }

    /// Shared tail of the path and byte v7 loaders. `path` is `None` for
    /// a byte image, which is not a sync destination.
    fn from_v7_load(mut l: crate::io_v7::V7Load, path: Option<&Path>) -> std::io::Result<Self> {
        let slot_to_id = std::mem::take(&mut l.ids);
        let inner = TurboQuantIndex::from_v7(l, path)?;

View on GitHub (pinned to ccab9f325e)

Solutions

1. Deduplicate or regenerate ids so every row has a unique external id before constructing the IdMap.
2. Use a stable unique key per document (e.g. hash of content + chunk index) instead of a reused counter.
3. Check duplicates first: sort a clone and compare adjacent elements to locate offending ids.

Example fix

// before
let map = IdMap::from_index_and_ids(index, ids)?; // Err: duplicate ids
// after
let mut sorted = ids.clone(); sorted.sort_unstable();
assert!(sorted.windows(2).all(|w| w[0] != w[1]), "duplicate ids");
let map = IdMap::from_index_and_ids(index, ids)?;

let mut s = ids.clone(); s.sort_unstable();
if s.windows(2).any(|w| w[0] == w[1]) {
    return Err("duplicate ids before IdMap build".into());
}

fn ids_unique(ids: &[u64]) -> bool {
    let mut s = ids.to_vec(); s.sort_unstable(); s.windows(2).all(|w| w[0] != w[1])
}

let map = IdMap::from_index_and_ids(index, ids)
    .map_err(|e| { log::error!("{e}"); BuildError::DuplicateIds })?;

When it happens

Trigger: Calling from_index_and_ids where slot_to_id contains the same u64 id at two or more slots — detected by sorted.windows(2).any(|w| w[0] == w[1]).

Common situations: Id-generation code reusing ids after removals; appending documents whose ids were already ingested; a buggy dedup step upstream that assigned the same id to multiple chunks.

Related errors

• {} ids for {} rows
• duplicate ids in v7 file
• invalid input value at vector {vi}, coord {ci}: {v} (must be
• duplicate id in batch: {k!r}
• {prefix} {version}; this turbovec accepts versions {list(com

AI-assisted analysis of RyanCodrai/turbovec@ccab9f325e (2026-09-06). Data as JSON: /api/errors/97ab64df322b73a2. Report an issue: GitHub.