MemPalace/mempalace · error · ValueError

add ids must be unique

Error message

add ids must be unique

What it means

Raised by QdrantCollection.add() when the ids list contains duplicates. add() is strict-insert semantics (unlike upsert); duplicate ids within one batch are rejected before any server call because points in a single Qdrant upsert batch with the same id would silently overwrite each other.

Source

Thrown at mempalace/backends/qdrant.py:824

            row
            for row in rows
            if (ids is None or row["id"] in set(ids))
            and _matches_where(row["metadata"], where)
            and _matches_where_document(row["document"], where_document)
        ]
        return rows

    def add(self, *, documents, ids, metadatas=None, embeddings=None):
        _validate_write_batch(
            documents=documents,
            ids=ids,
            metadatas=metadatas,
            embeddings=embeddings,
        )
        if embeddings is None:
            raise ValueError("qdrant requires explicit embeddings")
        if len(set(ids)) != len(ids):
            raise ValueError("add ids must be unique")
        existing = self.get(ids=list(ids), include=[])
        if existing.ids:
            raise ValueError(f"ids already exist in qdrant collection: {existing.ids}")
        self.upsert(documents=documents, ids=ids, metadatas=metadatas, embeddings=embeddings)

    def upsert(self, *, documents, ids, metadatas=None, embeddings=None):
        _validate_write_batch(
            documents=documents,
            ids=ids,
            metadatas=metadatas,
            embeddings=embeddings,
        )
        if embeddings is None:
            raise ValueError("qdrant requires explicit embeddings")
        vectors, dimension = _normalize_vectors(embeddings)
        self._ensure_remote_collection(dimension)
        metadatas = metadatas or [{} for _ in ids]
        points = []

View on GitHub (pinned to 06cb6987f0)

Solutions

1. Find the duplicate: from collections import Counter; [k for k,c in Counter(ids).items() if c>1]
2. Make ids unique by including a stable disambiguator (chunk index, uuid4, content hash + position)
3. If the duplicate rows are identical, deduplicate the batch before calling add
4. If you actually want to overwrite existing ids, call upsert() instead of add()

Example fix

# before
ids = [f"{doc_id}:0" for doc_id in doc_ids]  # repeated doc_ids -> duplicates
collection.add(documents=docs, ids=ids, embeddings=embs)
# after
ids = [f"{doc_id}:{i}" for doc_id, n in zip(doc_ids, counts) for i in range(n)]
collection.add(documents=docs, ids=ids, embeddings=embs)

from collections import Counter
dups = [k for k, c in Counter(ids).items() if c > 1]
if dups:
    raise ValueError(f"duplicate ids in batch: {dups}")

When it happens

Trigger: Calling add(ids=['a','b','a'], ...) — commonly when ids are generated from content hashes or chunk indices that collide, or when two loops concatenate batches and reuse id counters.

Common situations: Chunk-id generation that maps two chunks to the same hash (e.g. identical text chunks in one file); naive id scheme like f"{file}:{i}" where i resets per section; concatenated batches from parallel workers using overlapping id ranges.

Related errors

• ids already exist in qdrant collection: {existing.ids}
• update requires at least one of documents, metadatas, embedd
• {label} length {len(value)} does not match ids length {n}
• query requires query_embeddings
• Embedding model mismatch reading palace at {palace_path!r}.

AI-assisted analysis of MemPalace/mempalace@06cb6987f0 (2026-08-15). Data as JSON: /api/errors/3a9161eb5ab15b14. Report an issue: GitHub.