RyanCodrai/turbovec · error · ValueError

duplicate node_id {dup!r} appears multiple times in the inpu

Error message

duplicate node_id {dup!r} appears multiple times in the input batch; deduplicate before calling add()

What it means

add() refuses a batch containing two or more nodes with the same node_id. Duplicate ids would create ambiguous handle mappings in the quantized index, so the library fails fast via resolve_duplicates with DuplicatePolicy.REJECT instead of silently overwriting.

Source

Thrown at turbovec-python/python/turbovec/llama_index.py:355

        # below, so a generator / one-shot iterable would silently drain on
        # the first pass (async_add already does this via list(nodes)).
        nodes = list(nodes)
        if not nodes:
            return []

        # Reject intra-batch duplicates loudly. Letting them through would
        # leave the index with N vectors but only the last node_id mapped
        # back to one of them — the earlier handles become orphans that
        # `query` later resolves through the duplicate node_id, returning
        # the second node's payload attached to the first node's vector.
        # Caller's job to deduplicate before calling add.
        node_ids = [n.node_id for n in nodes]
        try:
            resolve_duplicates(node_ids, DuplicatePolicy.REJECT)
        except ValueError:
            seen: set[str] = set()
            dup = next(nid for nid in node_ids if nid in seen or seen.add(nid))
            raise ValueError(
                f"duplicate node_id {dup!r} appears multiple times "
                "in the input batch; deduplicate before calling add()"
            ) from None

        embeddings = [node.get_embedding() for node in nodes]
        vectors = np.asarray(embeddings, dtype=np.float32)
        if vectors.ndim != 2:
            raise ValueError(
                f"expected 2D embedding batch, got {vectors.ndim}D"
            )
        # A batch of empty per-node embeddings has shape (N, 0) — 2D, so
        # it passes the ndim guard, then dies deep in the index kernel
        # with an opaque buffer-length error. Name the real cause instead.
        if vectors.shape[1] == 0:
            raise ValueError(
                "nodes have empty embeddings (dim 0); check the embed "
                "model that produced them"
            )

View on GitHub (pinned to ccab9f325e)

Solutions

1. Deduplicate the batch before calling add(), e.g. {n.node_id: n for n in nodes}.values() or a seen-set filter
2. If re-insertion is intentional, delete the existing node first via delete_nodes
3. Assign unique ids (or let LlamaIndex auto-generate id_) when constructing nodes

Example fix

// before
store.add(nodes)
// after
unique = list({n.node_id: n for n in nodes}.values())
store.add(unique)

ids = [n.node_id for n in nodes]
assert len(ids) == len(set(ids)), f"duplicate node_ids in batch: {set(i for i in ids if ids.count(i)>1)}"

def all_unique_node_ids(nodes) -> bool:
    ids = [n.node_id for n in nodes]
    return len(ids) == len(set(ids))

try:
    store.add(nodes)
except ValueError as e:
    if "duplicate node_id" in str(e):
        nodes = list({n.node_id: n for n in nodes}.values())
        store.add(nodes)
    else:
        raise

When it happens

Trigger: Calling TurboQuantVectorStore.add(nodes) where two BaseNodes in the list share the same node_id (e.g. re-inserting the same node object, or building nodes with explicit id_ set to a constant).

Common situations: Re-running an ingestion pipeline without clearing the store; constructing nodes from rows where an id column repeats; accidentally appending the same node twice to a batch list.

Related errors

• TurboQuantVectorStore requires a pre-computed query_embeddin
• node embedding dim {vectors.shape[1]} does not match index d
• TurboQuantVectorStore.get(text_id) cannot return the origina
• filter condition {condition!r} not supported by TurboQuantVe
• filter operator {op!r} not supported by TurboQuantVectorStor

AI-assisted analysis of RyanCodrai/turbovec@ccab9f325e (2026-09-06). Data as JSON: /api/errors/3821a859bd2f5174. Report an issue: GitHub.