{"record":{"id":"3821a859bd2f5174","repo":"RyanCodrai/turbovec","slug":"duplicate-node-id-dup-r-appears-multiple-times-i","errorCode":null,"errorMessage":"duplicate node_id {dup!r} appears multiple times in the input batch; deduplicate before calling add()","messagePattern":"duplicate node_id (.+?) appears multiple times in the input batch; deduplicate before calling add\\(\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"turbovec-python/python/turbovec/llama_index.py","lineNumber":355,"sourceCode":"        # below, so a generator / one-shot iterable would silently drain on\n        # the first pass (async_add already does this via list(nodes)).\n        nodes = list(nodes)\n        if not nodes:\n            return []\n\n        # Reject intra-batch duplicates loudly. Letting them through would\n        # leave the index with N vectors but only the last node_id mapped\n        # back to one of them — the earlier handles become orphans that\n        # `query` later resolves through the duplicate node_id, returning\n        # the second node's payload attached to the first node's vector.\n        # Caller's job to deduplicate before calling add.\n        node_ids = [n.node_id for n in nodes]\n        try:\n            resolve_duplicates(node_ids, DuplicatePolicy.REJECT)\n        except ValueError:\n            seen: set[str] = set()\n            dup = next(nid for nid in node_ids if nid in seen or seen.add(nid))\n            raise ValueError(\n                f\"duplicate node_id {dup!r} appears multiple times \"\n                \"in the input batch; deduplicate before calling add()\"\n            ) from None\n\n        embeddings = [node.get_embedding() for node in nodes]\n        vectors = np.asarray(embeddings, dtype=np.float32)\n        if vectors.ndim != 2:\n            raise ValueError(\n                f\"expected 2D embedding batch, got {vectors.ndim}D\"\n            )\n        # A batch of empty per-node embeddings has shape (N, 0) — 2D, so\n        # it passes the ndim guard, then dies deep in the index kernel\n        # with an opaque buffer-length error. Name the real cause instead.\n        if vectors.shape[1] == 0:\n            raise ValueError(\n                \"nodes have empty embeddings (dim 0); check the embed \"\n                \"model that produced them\"\n            )","sourceCodeStart":337,"sourceCodeEnd":373,"githubUrl":"https://github.com/RyanCodrai/turbovec/blob/ccab9f325e6ce2a270a87daf01ae4e443bcf2d49/turbovec-python/python/turbovec/llama_index.py#L337-L373","documentation":"add() refuses a batch containing two or more nodes with the same node_id. Duplicate ids would create ambiguous handle mappings in the quantized index, so the library fails fast via resolve_duplicates with DuplicatePolicy.REJECT instead of silently overwriting.","triggerScenarios":"Calling TurboQuantVectorStore.add(nodes) where two BaseNodes in the list share the same node_id (e.g. re-inserting the same node object, or building nodes with explicit id_ set to a constant).","commonSituations":"Re-running an ingestion pipeline without clearing the store; constructing nodes from rows where an id column repeats; accidentally appending the same node twice to a batch list.","solutions":["Deduplicate the batch before calling add(), e.g. {n.node_id: n for n in nodes}.values() or a seen-set filter","If re-insertion is intentional, delete the existing node first via delete_nodes","Assign unique ids (or let LlamaIndex auto-generate id_) when constructing nodes"],"exampleFix":"// before\nstore.add(nodes)\n// after\nunique = list({n.node_id: n for n in nodes}.values())\nstore.add(unique)","handlingStrategy":"validation","validationCode":"ids = [n.node_id for n in nodes]\nassert len(ids) == len(set(ids)), f\"duplicate node_ids in batch: {set(i for i in ids if ids.count(i)>1)}\"","typeGuard":"def all_unique_node_ids(nodes) -> bool:\n    ids = [n.node_id for n in nodes]\n    return len(ids) == len(set(ids))","tryCatchPattern":"try:\n    store.add(nodes)\nexcept ValueError as e:\n    if \"duplicate node_id\" in str(e):\n        nodes = list({n.node_id: n for n in nodes}.values())\n        store.add(nodes)\n    else:\n        raise","preventionTips":["Deduplicate batches by node_id before every add()","Let LlamaIndex auto-generate node ids instead of hardcoding id_","Make ingestion pipelines idempotent by deleting before re-inserting"],"tags":["python","llama-index","vector-store","duplicate-id"],"backgroundTag":"duplicate-identifier","analyzedSha":"ccab9f325e6ce2a270a87daf01ae4e443bcf2d49","analyzedAt":"2026-09-06T08:39:18.516Z","contentChangedAt":"2026-09-06T08:39:18.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}