deepset-ai/haystack · error · DuplicateDocumentError
ID '{document.id}' already exists.
Error message
ID '{document.id}' already exists. What it means
InMemoryDocumentStore.write_documents raises DuplicateDocumentError when a document with the same ID already exists in storage and the effective DuplicatePolicy is FAIL (the default). The store refuses silent overwrites so callers consciously choose how duplicates are handled.
Source
Thrown at haystack/document_stores/in_memory/document_store.py:482
Refer to the DocumentStore.write_documents() protocol documentation.
If `policy` is set to `DuplicatePolicy.NONE` defaults to `DuplicatePolicy.FAIL`.
"""
if (
not isinstance(documents, Iterable)
or isinstance(documents, str)
or any(not isinstance(doc, Document) for doc in documents)
):
raise ValueError("Please provide a list of Documents.")
if policy == DuplicatePolicy.NONE:
policy = DuplicatePolicy.FAIL
written_documents = len(documents)
for document in documents:
if policy != DuplicatePolicy.OVERWRITE and document.id in self.storage.keys():
if policy == DuplicatePolicy.FAIL:
raise DuplicateDocumentError(f"ID '{document.id}' already exists.")
if policy == DuplicatePolicy.SKIP:
logger.warning("ID '{document_id}' already exists", document_id=document.id)
written_documents -= 1
continue
# Since the statistics are updated in an incremental manner,
# we need to explicitly remove the existing document to revert
# the statistics before updating them with the new document.
if document.id in self.storage.keys():
self.delete_documents([document.id])
tokens = []
if document.content is not None:
tokens = self._tokenize_bm25(document.content)
self.storage[document.id] = document
self._bm25_attr[document.id] = BM25DocumentStats(Counter(tokens), len(tokens))View on GitHub (pinned to e318778c9b)
Solutions
- Pass policy=DuplicatePolicy.OVERWRITE to replace existing documents
- Pass policy=DuplicatePolicy.SKIP to keep existing documents and skip duplicates
- Use policy=DuplicatePolicy.APPEND (new IDs only) if IDs are expected to be unique
- Delete the conflicting documents first (delete_documents/delete_all_documents) before rewriting
Example fix
// before store.write_documents(documents) // after from haystack.document_stores.types import DuplicatePolicy store.write_documents(documents, policy=DuplicatePolicy.OVERWRITE)
Defensive patterns
Strategy: try-catch
Validate before calling
from haystack.document_stores.types import DuplicatePolicy
existing = {d.id for d in store.filter_documents()}
conflicts = [d.id for d in documents if d.id in existing]
if conflicts:
documents = [d for d in documents if d.id not in conflicts] # or plan to overwrite Try / catch
from haystack.document_stores.errors import DuplicateDocumentError
try:
store.write_documents(documents)
except DuplicateDocumentError:
store.write_documents(documents, policy=DuplicatePolicy.OVERWRITE) Prevention
- Always pass an explicit DuplicatePolicy instead of relying on the FAIL default
- Use deterministic content-based IDs so re-indexing is idempotent with OVERWRITE
- For re-runnable pipelines, start from a fresh store or delete documents first
When it happens
Trigger: Calling write_documents(documents) (or write_documents_async) twice with the same document IDs without passing policy=DuplicatePolicy.OVERWRITE/SKIP; re-running a pipeline that re-indexes the same documents into a non-fresh store.
Common situations: Re-running indexing scripts or notebooks against a persisted store, re-ingesting a file whose documents get deterministic IDs, forgetting that DuplicatePolicy.FAIL is the default.
Related errors
- Invalid filter syntax. See https://docs.haystack.deepset.ai/
- tools must be a list of Tool and/or Toolset objects, a Tools
- StateSchema: Key '{param}' is missing a 'type' entry.
- StateSchema: 'type' for key '{param}' must be a Python type,
- StateSchema: 'handler' for key '{param}' must be callable or
AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30).
Data as JSON: /api/errors/93c55804aa4ef654.
Report an issue: GitHub.