{"record":{"id":"5ebd9c6ffd95b0c1","repo":"MemPalace/mempalace","slug":"ids-already-exist-in-qdrant-collection-existing","errorCode":null,"errorMessage":"ids already exist in qdrant collection: {existing.ids}","messagePattern":"ids already exist in qdrant collection: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"mempalace/backends/qdrant.py","lineNumber":827,"sourceCode":"            and _matches_where(row[\"metadata\"], where)\n            and _matches_where_document(row[\"document\"], where_document)\n        ]\n        return rows\n\n    def add(self, *, documents, ids, metadatas=None, embeddings=None):\n        _validate_write_batch(\n            documents=documents,\n            ids=ids,\n            metadatas=metadatas,\n            embeddings=embeddings,\n        )\n        if embeddings is None:\n            raise ValueError(\"qdrant requires explicit embeddings\")\n        if len(set(ids)) != len(ids):\n            raise ValueError(\"add ids must be unique\")\n        existing = self.get(ids=list(ids), include=[])\n        if existing.ids:\n            raise ValueError(f\"ids already exist in qdrant collection: {existing.ids}\")\n        self.upsert(documents=documents, ids=ids, metadatas=metadatas, embeddings=embeddings)\n\n    def upsert(self, *, documents, ids, metadatas=None, embeddings=None):\n        _validate_write_batch(\n            documents=documents,\n            ids=ids,\n            metadatas=metadatas,\n            embeddings=embeddings,\n        )\n        if embeddings is None:\n            raise ValueError(\"qdrant requires explicit embeddings\")\n        vectors, dimension = _normalize_vectors(embeddings)\n        self._ensure_remote_collection(dimension)\n        metadatas = metadatas or [{} for _ in ids]\n        points = []\n        for doc_id, doc, meta, vector in zip(ids, documents, metadatas, vectors):\n            points.append(\n                {","sourceCodeStart":809,"sourceCodeEnd":845,"githubUrl":"https://github.com/MemPalace/mempalace/blob/06cb6987f02610784fefbad4b2bd5d026d164ba6/mempalace/backends/qdrant.py#L809-L845","documentation":"Raised by QdrantCollection.add() when some of the requested ids already exist in the collection (checked via a get() pre-flight). add() is insert-only; overwriting existing points requires upsert(). The message includes the offending ids.","triggerScenarios":"Calling add() with ids that were stored in a previous run — typical in retry logic after a partial failure, or idempotent re-ingest of the same transcript/file without a freshness check.","commonSituations":"Re-running an ingest pipeline after a crash; a hook firing twice for the same session; deterministic chunk ids colliding with yesterday's data; duplicate file processing.","solutions":["Switch to upsert() if overwriting is the intended behavior (idempotent re-ingest)","Or skip existing ids: filter out the ones named in the error before retrying add()","If duplicate ingest indicates a bug (same content filed twice), fix the upstream trigger (e.g. hook firing twice) instead of the write path","Keep a manifest of ingested ids per source to make re-ingest checks cheap"],"exampleFix":"# before\ncollection.add(documents=docs, ids=ids, embeddings=embs)  # ValueError: ids already exist\n// after\nexisting = set(collection.get(ids=ids, include=[]).ids)\nnew = [(d,i,e) for d,i,e in zip(docs,ids,embs) if i not in existing]\nif new:\n    collection.upsert(documents=[d for d,_,_ in new], ids=[i for _,i,_ in new], embeddings=[e for _,_,e in new])","handlingStrategy":"validation","validationCode":"existing = set(collection.get(ids=ids, include=[]).ids)\nfresh = [(d, i, e) for d, i, e in zip(docs, ids, embs) if i not in existing]\nif fresh:\n    collection.upsert(*[list(x) for x in zip(*fresh)])","typeGuard":null,"tryCatchPattern":"try:\n    collection.add(documents=docs, ids=ids, embeddings=embs)\nexcept ValueError as e:\n    if \"already exist\" in str(e):\n        collection.upsert(documents=docs, ids=ids, embeddings=embs)  # idempotent retry","preventionTips":["Make ingest idempotent: use upsert() for re-runnable pipelines","Track ingested ids per source in a manifest to skip cheaply","Watch hooks for double-fire — same-session double ingest is the usual root cause"],"tags":["ids","idempotency","qdrant","api-misuse"],"backgroundTag":null,"analyzedSha":"06cb6987f02610784fefbad4b2bd5d026d164ba6","analyzedAt":"2026-08-15T03:03:36.213Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}