RyanCodrai/turbovec · error · ValueError

duplicate id in batch

Error message

duplicate id in batch: {k!r}

What it means

DuplicatePolicy.REJECT refuses to store a batch containing two entries with the same id. resolve_duplicates scans the keys while computing the keep-mask and raises ValueError on the second occurrence of any id, because REJECT means the whole batch should be rejected rather than silently collapsing duplicates.

Solutions

1. Deduplicate the batch before the call: `keys = list(dict.fromkeys(keys))` (keeps first occurrence) or dedupe the full records.
2. Choose a different policy if collapsing is acceptable: DuplicatePolicy.KEEP_FIRST, KEEP_LAST, or KEEP_ALL.
3. Wrap in try/except ValueError and report/handle the offending key (it is quoted in the message).
4. Fix the upstream producer so ids are unique per batch (e.g. enforce uniqueness at ingest).

Example fix

// before
store.add(ids=['a', 'b', 'a'], policy=DuplicatePolicy.REJECT)
// after
ids = list(dict.fromkeys(['a', 'b', 'a']))  # -> ['a', 'b']
store.add(ids=ids, policy=DuplicatePolicy.REJECT)

from turbovec._dedup import DuplicatePolicy
def ensure_unique(keys):
    seen = set()
    for k in keys:
        if k in seen:
            raise ValueError(f'duplicate id in batch: {k!r}')
        seen.add(k)
ensure_unique(ids)

def all_unique(keys) -> bool:
    return len(keys) == len(set(keys))

try:
    store.add(ids=ids, policy=DuplicatePolicy.REJECT)
except ValueError as e:
    if 'duplicate id in batch' in str(e):
        ids = list(dict.fromkeys(ids))  # dedupe and retry
    else:
        raise

When it happens

Trigger: Calling `add`, `add_texts`-style storage (_store_texts_and_vectors), or `resolve_duplicates(keys, policy=DuplicatePolicy.REJECT)` with a list of ids/keys containing at least one repeated value, e.g. resolve_duplicates(['a','b','a'], DuplicatePolicy.REJECT).

Common situations: Batching user-submitted records where ids are not deduplicated upstream; concatenating chunks from multiple sources that overlap; using REJECT policy as an integrity check on data you expected to be unique.

Related errors

• nodes have empty embeddings (dim 0); check the embed model…
• ; this turbovec accepts versions
• bit_width must be 2, 3, or 4, got
• Both metadata value and filter value must be strings for…
• Both metadata value and filter value must be strings for…

AI-assisted analysis of RyanCodrai/turbovec@ccab9f325e (2026-09-06). Data as JSON: /api/errors/ce58721c467898cc. Report an issue: GitHub.

Appendix: source

Thrown at turbovec-python/python/turbovec/_dedup.py:60

def resolve_duplicates(
    keys: Sequence[Hashable], policy: DuplicatePolicy
) -> List[int]:
    """Return, in ascending order, the batch indices to keep under ``policy``.

    The returned indices index into ``keys`` (and any parallel arrays the
    caller holds). For KEEP_ALL and REJECT the result is ``0..len(keys)``;
    for KEEP_LAST/KEEP_FIRST it collapses to one index per distinct key.

    Raises:
        ValueError: under REJECT, if any key occurs more than once.
    """
    if policy is DuplicatePolicy.KEEP_ALL:
        return list(range(len(keys)))
    if policy is DuplicatePolicy.REJECT:
        seen: set = set()
        for k in keys:
            if k in seen:
                raise ValueError(f"duplicate id in batch: {k!r}")
            seen.add(k)
        return list(range(len(keys)))
    # KEEP_LAST / KEEP_FIRST collapse to one index per key.
    chosen: dict = {}
    for i, k in enumerate(keys):
        if policy is DuplicatePolicy.KEEP_LAST or k not in chosen:
            chosen[k] = i
    return sorted(chosen.values())


__all__ = ["DuplicatePolicy", "resolve_duplicates"]

View on GitHub (pinned to ccab9f325e)