RyanCodrai/turbovec · error · ValueError

duplicate id in batch: {k!r}

Error message

duplicate id in batch: {k!r}

What it means

DuplicatePolicy.REJECT refuses to store a batch containing two entries with the same id. resolve_duplicates scans the keys while computing the keep-mask and raises ValueError on the second occurrence of any id, because REJECT means the whole batch should be rejected rather than silently collapsing duplicates.

Source

Thrown at turbovec-python/python/turbovec/_dedup.py:60

def resolve_duplicates(
    keys: Sequence[Hashable], policy: DuplicatePolicy
) -> List[int]:
    """Return, in ascending order, the batch indices to keep under ``policy``.

    The returned indices index into ``keys`` (and any parallel arrays the
    caller holds). For KEEP_ALL and REJECT the result is ``0..len(keys)``;
    for KEEP_LAST/KEEP_FIRST it collapses to one index per distinct key.

    Raises:
        ValueError: under REJECT, if any key occurs more than once.
    """
    if policy is DuplicatePolicy.KEEP_ALL:
        return list(range(len(keys)))
    if policy is DuplicatePolicy.REJECT:
        seen: set = set()
        for k in keys:
            if k in seen:
                raise ValueError(f"duplicate id in batch: {k!r}")
            seen.add(k)
        return list(range(len(keys)))
    # KEEP_LAST / KEEP_FIRST collapse to one index per key.
    chosen: dict = {}
    for i, k in enumerate(keys):
        if policy is DuplicatePolicy.KEEP_LAST or k not in chosen:
            chosen[k] = i
    return sorted(chosen.values())


__all__ = ["DuplicatePolicy", "resolve_duplicates"]

View on GitHub (pinned to ccab9f325e)

Solutions

1. Deduplicate the batch before the call: `keys = list(dict.fromkeys(keys))` (keeps first occurrence) or dedupe the full records.
2. Choose a different policy if collapsing is acceptable: DuplicatePolicy.KEEP_FIRST, KEEP_LAST, or KEEP_ALL.
3. Wrap in try/except ValueError and report/handle the offending key (it is quoted in the message).
4. Fix the upstream producer so ids are unique per batch (e.g. enforce uniqueness at ingest).

Example fix

// before
store.add(ids=['a', 'b', 'a'], policy=DuplicatePolicy.REJECT)
// after
ids = list(dict.fromkeys(['a', 'b', 'a']))  # -> ['a', 'b']
store.add(ids=ids, policy=DuplicatePolicy.REJECT)

from turbovec._dedup import DuplicatePolicy
def ensure_unique(keys):
    seen = set()
    for k in keys:
        if k in seen:
            raise ValueError(f'duplicate id in batch: {k!r}')
        seen.add(k)
ensure_unique(ids)

def all_unique(keys) -> bool:
    return len(keys) == len(set(keys))

try:
    store.add(ids=ids, policy=DuplicatePolicy.REJECT)
except ValueError as e:
    if 'duplicate id in batch' in str(e):
        ids = list(dict.fromkeys(ids))  # dedupe and retry
    else:
        raise

When it happens

Trigger: Calling `add`, `add_texts`-style storage (_store_texts_and_vectors), or `resolve_duplicates(keys, policy=DuplicatePolicy.REJECT)` with a list of ids/keys containing at least one repeated value, e.g. resolve_duplicates(['a','b','a'], DuplicatePolicy.REJECT).

Common situations: Batching user-submitted records where ids are not deduplicated upstream; concatenating chunks from multiple sources that overlap; using REJECT policy as an integrity check on data you expected to be unique.

Related errors

• {prefix} {version}; this turbovec accepts versions {list(com
• nodes have empty embeddings (dim 0); check the embed model t
• module {__name__!r} has no attribute {name!r}
• side-car key {key!r} at {_crumb_path(entry)} is {type(key)._
• side-car value at {_crumb_path(entry)} is {obj!r}, which JSO

AI-assisted analysis of RyanCodrai/turbovec@ccab9f325e (2026-09-06). Data as JSON: /api/errors/ce58721c467898cc. Report an issue: GitHub.