{"record":{"id":"3a9161eb5ab15b14","repo":"MemPalace/mempalace","slug":"add-ids-must-be-unique-3a9161","errorCode":null,"errorMessage":"add ids must be unique","messagePattern":"add ids must be unique","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"mempalace/backends/qdrant.py","lineNumber":824,"sourceCode":"            row\n            for row in rows\n            if (ids is None or row[\"id\"] in set(ids))\n            and _matches_where(row[\"metadata\"], where)\n            and _matches_where_document(row[\"document\"], where_document)\n        ]\n        return rows\n\n    def add(self, *, documents, ids, metadatas=None, embeddings=None):\n        _validate_write_batch(\n            documents=documents,\n            ids=ids,\n            metadatas=metadatas,\n            embeddings=embeddings,\n        )\n        if embeddings is None:\n            raise ValueError(\"qdrant requires explicit embeddings\")\n        if len(set(ids)) != len(ids):\n            raise ValueError(\"add ids must be unique\")\n        existing = self.get(ids=list(ids), include=[])\n        if existing.ids:\n            raise ValueError(f\"ids already exist in qdrant collection: {existing.ids}\")\n        self.upsert(documents=documents, ids=ids, metadatas=metadatas, embeddings=embeddings)\n\n    def upsert(self, *, documents, ids, metadatas=None, embeddings=None):\n        _validate_write_batch(\n            documents=documents,\n            ids=ids,\n            metadatas=metadatas,\n            embeddings=embeddings,\n        )\n        if embeddings is None:\n            raise ValueError(\"qdrant requires explicit embeddings\")\n        vectors, dimension = _normalize_vectors(embeddings)\n        self._ensure_remote_collection(dimension)\n        metadatas = metadatas or [{} for _ in ids]\n        points = []","sourceCodeStart":806,"sourceCodeEnd":842,"githubUrl":"https://github.com/MemPalace/mempalace/blob/06cb6987f02610784fefbad4b2bd5d026d164ba6/mempalace/backends/qdrant.py#L806-L842","documentation":"Raised by QdrantCollection.add() when the ids list contains duplicates. add() is strict-insert semantics (unlike upsert); duplicate ids within one batch are rejected before any server call because points in a single Qdrant upsert batch with the same id would silently overwrite each other.","triggerScenarios":"Calling add(ids=['a','b','a'], ...) — commonly when ids are generated from content hashes or chunk indices that collide, or when two loops concatenate batches and reuse id counters.","commonSituations":"Chunk-id generation that maps two chunks to the same hash (e.g. identical text chunks in one file); naive id scheme like f\"{file}:{i}\" where i resets per section; concatenated batches from parallel workers using overlapping id ranges.","solutions":["Find the duplicate: from collections import Counter; [k for k,c in Counter(ids).items() if c>1]","Make ids unique by including a stable disambiguator (chunk index, uuid4, content hash + position)","If the duplicate rows are identical, deduplicate the batch before calling add","If you actually want to overwrite existing ids, call upsert() instead of add()"],"exampleFix":"# before\nids = [f\"{doc_id}:0\" for doc_id in doc_ids]  # repeated doc_ids -> duplicates\ncollection.add(documents=docs, ids=ids, embeddings=embs)\n# after\nids = [f\"{doc_id}:{i}\" for doc_id, n in zip(doc_ids, counts) for i in range(n)]\ncollection.add(documents=docs, ids=ids, embeddings=embs)","handlingStrategy":"validation","validationCode":"from collections import Counter\ndups = [k for k, c in Counter(ids).items() if c > 1]\nif dups:\n    raise ValueError(f\"duplicate ids in batch: {dups}\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Generate ids from (source, chunk-index, content-hash) triples so collisions are meaningful and rare","Deduplicate batches before writes when sources may overlap","Prefer upsert() when overwrite semantics are acceptable"],"tags":["validation","ids","qdrant","api-misuse"],"backgroundTag":null,"analyzedSha":"06cb6987f02610784fefbad4b2bd5d026d164ba6","analyzedAt":"2026-08-15T03:03:36.213Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}