{"record":{"id":"ce58721c467898cc","repo":"RyanCodrai/turbovec","slug":"duplicate-id-in-batch-k-r","errorCode":null,"errorMessage":"duplicate id in batch: {k!r}","messagePattern":"duplicate id in batch: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"turbovec-python/python/turbovec/_dedup.py","lineNumber":60,"sourceCode":"def resolve_duplicates(\n    keys: Sequence[Hashable], policy: DuplicatePolicy\n) -> List[int]:\n    \"\"\"Return, in ascending order, the batch indices to keep under ``policy``.\n\n    The returned indices index into ``keys`` (and any parallel arrays the\n    caller holds). For KEEP_ALL and REJECT the result is ``0..len(keys)``;\n    for KEEP_LAST/KEEP_FIRST it collapses to one index per distinct key.\n\n    Raises:\n        ValueError: under REJECT, if any key occurs more than once.\n    \"\"\"\n    if policy is DuplicatePolicy.KEEP_ALL:\n        return list(range(len(keys)))\n    if policy is DuplicatePolicy.REJECT:\n        seen: set = set()\n        for k in keys:\n            if k in seen:\n                raise ValueError(f\"duplicate id in batch: {k!r}\")\n            seen.add(k)\n        return list(range(len(keys)))\n    # KEEP_LAST / KEEP_FIRST collapse to one index per key.\n    chosen: dict = {}\n    for i, k in enumerate(keys):\n        if policy is DuplicatePolicy.KEEP_LAST or k not in chosen:\n            chosen[k] = i\n    return sorted(chosen.values())\n\n\n__all__ = [\"DuplicatePolicy\", \"resolve_duplicates\"]\n","sourceCodeStart":42,"sourceCodeEnd":72,"githubUrl":"https://github.com/RyanCodrai/turbovec/blob/ccab9f325e6ce2a270a87daf01ae4e443bcf2d49/turbovec-python/python/turbovec/_dedup.py#L42-L72","documentation":"DuplicatePolicy.REJECT refuses to store a batch containing two entries with the same id. resolve_duplicates scans the keys while computing the keep-mask and raises ValueError on the second occurrence of any id, because REJECT means the whole batch should be rejected rather than silently collapsing duplicates.","triggerScenarios":"Calling `add`, `add_texts`-style storage (_store_texts_and_vectors), or `resolve_duplicates(keys, policy=DuplicatePolicy.REJECT)` with a list of ids/keys containing at least one repeated value, e.g. resolve_duplicates(['a','b','a'], DuplicatePolicy.REJECT).","commonSituations":"Batching user-submitted records where ids are not deduplicated upstream; concatenating chunks from multiple sources that overlap; using REJECT policy as an integrity check on data you expected to be unique.","solutions":["Deduplicate the batch before the call: `keys = list(dict.fromkeys(keys))` (keeps first occurrence) or dedupe the full records.","Choose a different policy if collapsing is acceptable: DuplicatePolicy.KEEP_FIRST, KEEP_LAST, or KEEP_ALL.","Wrap in try/except ValueError and report/handle the offending key (it is quoted in the message).","Fix the upstream producer so ids are unique per batch (e.g. enforce uniqueness at ingest)."],"exampleFix":"// before\nstore.add(ids=['a', 'b', 'a'], policy=DuplicatePolicy.REJECT)\n// after\nids = list(dict.fromkeys(['a', 'b', 'a']))  # -> ['a', 'b']\nstore.add(ids=ids, policy=DuplicatePolicy.REJECT)","handlingStrategy":"validation","validationCode":"from turbovec._dedup import DuplicatePolicy\ndef ensure_unique(keys):\n    seen = set()\n    for k in keys:\n        if k in seen:\n            raise ValueError(f'duplicate id in batch: {k!r}')\n        seen.add(k)\nensure_unique(ids)","typeGuard":"def all_unique(keys) -> bool:\n    return len(keys) == len(set(keys))","tryCatchPattern":"try:\n    store.add(ids=ids, policy=DuplicatePolicy.REJECT)\nexcept ValueError as e:\n    if 'duplicate id in batch' in str(e):\n        ids = list(dict.fromkeys(ids))  # dedupe and retry\n    else:\n        raise","preventionTips":["Deduplicate ids at ingestion time, before batching.","Enforce unique ids upstream (database primary key / set-based ingest).","Use KEEP_FIRST/KEEP_LAST when collapsing duplicates is acceptable.","Write a unit test asserting batch ids are unique before add()."],"tags":["python","validation","duplicates","batch"],"backgroundTag":"duplicate-key","analyzedSha":"ccab9f325e6ce2a270a87daf01ae4e443bcf2d49","analyzedAt":"2026-09-06T08:39:18.516Z","contentChangedAt":"2026-09-06T08:39:18.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}