{"record":{"id":"93c55804aa4ef654","repo":"deepset-ai/haystack","slug":"id-document-id-already-exists","errorCode":null,"errorMessage":"ID '{document.id}' already exists.","messagePattern":"ID '(.+?)' already exists\\.","errorType":"exception","errorClass":"DuplicateDocumentError","httpStatus":null,"severity":"error","filePath":"haystack/document_stores/in_memory/document_store.py","lineNumber":482,"sourceCode":"        Refer to the DocumentStore.write_documents() protocol documentation.\n\n        If `policy` is set to `DuplicatePolicy.NONE` defaults to `DuplicatePolicy.FAIL`.\n        \"\"\"\n        if (\n            not isinstance(documents, Iterable)\n            or isinstance(documents, str)\n            or any(not isinstance(doc, Document) for doc in documents)\n        ):\n            raise ValueError(\"Please provide a list of Documents.\")\n\n        if policy == DuplicatePolicy.NONE:\n            policy = DuplicatePolicy.FAIL\n\n        written_documents = len(documents)\n        for document in documents:\n            if policy != DuplicatePolicy.OVERWRITE and document.id in self.storage.keys():\n                if policy == DuplicatePolicy.FAIL:\n                    raise DuplicateDocumentError(f\"ID '{document.id}' already exists.\")\n                if policy == DuplicatePolicy.SKIP:\n                    logger.warning(\"ID '{document_id}' already exists\", document_id=document.id)\n                    written_documents -= 1\n                    continue\n\n            # Since the statistics are updated in an incremental manner,\n            # we need to explicitly remove the existing document to revert\n            # the statistics before updating them with the new document.\n            if document.id in self.storage.keys():\n                self.delete_documents([document.id])\n\n            tokens = []\n            if document.content is not None:\n                tokens = self._tokenize_bm25(document.content)\n\n            self.storage[document.id] = document\n\n            self._bm25_attr[document.id] = BM25DocumentStats(Counter(tokens), len(tokens))","sourceCodeStart":464,"sourceCodeEnd":500,"githubUrl":"https://github.com/deepset-ai/haystack/blob/e318778c9bf60a1963e3b5f451359655dd696c30/haystack/document_stores/in_memory/document_store.py#L464-L500","documentation":"InMemoryDocumentStore.write_documents raises DuplicateDocumentError when a document with the same ID already exists in storage and the effective DuplicatePolicy is FAIL (the default). The store refuses silent overwrites so callers consciously choose how duplicates are handled.","triggerScenarios":"Calling write_documents(documents) (or write_documents_async) twice with the same document IDs without passing policy=DuplicatePolicy.OVERWRITE/SKIP; re-running a pipeline that re-indexes the same documents into a non-fresh store.","commonSituations":"Re-running indexing scripts or notebooks against a persisted store, re-ingesting a file whose documents get deterministic IDs, forgetting that DuplicatePolicy.FAIL is the default.","solutions":["Pass policy=DuplicatePolicy.OVERWRITE to replace existing documents","Pass policy=DuplicatePolicy.SKIP to keep existing documents and skip duplicates","Use policy=DuplicatePolicy.APPEND (new IDs only) if IDs are expected to be unique","Delete the conflicting documents first (delete_documents/delete_all_documents) before rewriting"],"exampleFix":"// before\nstore.write_documents(documents)\n// after\nfrom haystack.document_stores.types import DuplicatePolicy\nstore.write_documents(documents, policy=DuplicatePolicy.OVERWRITE)","handlingStrategy":"try-catch","validationCode":"from haystack.document_stores.types import DuplicatePolicy\nexisting = {d.id for d in store.filter_documents()}\nconflicts = [d.id for d in documents if d.id in existing]\nif conflicts:\n    documents = [d for d in documents if d.id not in conflicts]  # or plan to overwrite","typeGuard":null,"tryCatchPattern":"from haystack.document_stores.errors import DuplicateDocumentError\ntry:\n    store.write_documents(documents)\nexcept DuplicateDocumentError:\n    store.write_documents(documents, policy=DuplicatePolicy.OVERWRITE)","preventionTips":["Always pass an explicit DuplicatePolicy instead of relying on the FAIL default","Use deterministic content-based IDs so re-indexing is idempotent with OVERWRITE","For re-runnable pipelines, start from a fresh store or delete documents first"],"tags":["document-store","duplicate-id","python"],"backgroundTag":"duplicate-key-conflict","analyzedSha":"e318778c9bf60a1963e3b5f451359655dd696c30","analyzedAt":"2026-08-30T11:45:20.711Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}