RyanCodrai/turbovec · error · DuplicateDocumentError

ID '{doc.id}' already exists in the document store.

Error message

ID '{doc.id}' already exists in the document store.

What it means

DuplicateDocumentError raised in _write_documents_locked when policy is FAIL (the default: NONE resolves to FAIL) and a document's id already exists in the store. Committed per document in iteration order, matching InMemoryDocumentStore's partial-write semantics: documents before the collision stay persisted.

Source

Thrown at turbovec-python/python/turbovec/haystack.py:262

        if policy == DuplicatePolicy.FAIL:
            # Reference parity (issue #167): InMemoryDocumentStore commits
            # each document as it iterates and raises on the *first*
            # duplicate, so every non-duplicate document preceding it
            # stays persisted — a partial write. Mirror that observable
            # state exactly: validate and commit per document, raising on
            # the first collision. A repeated id within a single call
            # collides with its already-committed first instance, the same
            # way a cross-call repeat would. Each individual commit is
            # still all-or-nothing — validation precedes any mutation, so
            # a failing document mid-batch never leaves the index and the
            # id maps inconsistent (#89/#139 apply per document).
            written = 0
            for doc in documents:
                if doc.id in self._str_to_u64:
                    # Checked before embedding validation: the reference
                    # raises DuplicateDocumentError for a colliding id
                    # regardless of the document's other fields.
                    raise DuplicateDocumentError(
                        f"ID '{doc.id}' already exists in the document store."
                    )
                if doc.embedding is None:
                    raise ValueError(
                        f"Document {doc.id!r} has no embedding. "
                        "TurboQuantDocumentStore only stores documents with precomputed "
                        "embeddings — run an embedder component before writing."
                    )
                self._commit_batch([doc])
                written += 1
            return written

        # SKIP / OVERWRITE: first pass validates and resolves duplicates
        # against the batch-so-far as well as the existing store:
        # InMemoryDocumentStore writes into its dict as it iterates, so a
        # repeated id *within a single call* is resolved the same way a
        # cross-call repeat would be. Without tracking the batch, every
        # duplicate row still gets its own vector while _str_to_u64 keeps

View on GitHub (pinned to ccab9f325e)

Solutions

1. Pass policy=DuplicatePolicy.SKIP to silently skip existing ids, or OVERWRITE to replace them.
2. Deduplicate the batch against existing ids (filter_documents / count_documents) before writing.
3. Catch DuplicateDocumentError and retry the remaining documents under a non-FAIL policy.

When it happens

Trigger: Thrown at turbovec-python/python/turbovec/haystack.py:262 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of RyanCodrai/turbovec@ccab9f325e (2026-09-06). Data as JSON: /api/errors/85b8ca1e0f18de1a. Report an issue: GitHub.