RyanCodrai/turbovec · error · ValueError

persisted store is corrupt: duplicate document ids in the si

Error message

persisted store is corrupt: duplicate document ids in the side-car

What it means

Raised in load_from_disk after rebuilding _str_to_u64 from the reloaded doc table: the rebuilt map is shorter than the handle table, meaning two handles shared one document id. The write path enforces unique ids, so duplicates can only come from a hand-edited or otherwise corrupt docstore.json side-car; left unchecked, a duplicate would silently collapse into a shadow document that is searchable but unreachable and undeletable by id.

Source

Thrown at turbovec-python/python/turbovec/haystack.py:920

        store._index = IdMapIndex.load(str(folder / "index.tvim"))
        # Reconstruct {int handle: doc data} from the list-of-pairs form.
        # `_deserialize_doc_data` is shape-tolerant: v1 entries lack the
        # `blob` / `sparse_embedding` keys and come back with both set to
        # None, which matches their original on-write state.
        store._u64_to_doc = {
            int(h): cls._deserialize_doc_data(d) for h, d in state["u64_to_doc"]
        }
        store._next_u64 = state["next_u64"]
        # Rebuild str_to_u64 from the reloaded doc table.
        store._str_to_u64 = {
            data["id"]: handle for handle, data in store._u64_to_doc.items()
        }
        # Two handles sharing a document id would silently collapse in the
        # rebuild above, leaving a shadow document that is searchable but
        # unreachable (and undeletable) by id. The write path enforces
        # unique ids, so a duplicate can only mean a corrupt side-car.
        if len(store._str_to_u64) != len(store._u64_to_doc):
            raise ValueError(
                "persisted store is corrupt: duplicate document ids in the side-car"
            )
        check_persisted_handles(
            store._index,
            store._u64_to_doc.keys(),
            what="document",
            next_u64=store._next_u64,
        )
        return store

    # ---- Copy & pickle ------------------------------------------------
    #
    # The Rust index is not directly picklable; it round-trips through
    # the core's in-memory ``.tvim`` byte format
    # (``IdMapIndex.to_bytes`` / ``from_bytes``). The per-store lock and
    # the async executor are excluded from the state — neither can cross
    # pickling — and recreated on restore; a restored/copied store always
    # owns a fresh executor, even when the original wrapped a

View on GitHub (pinned to ccab9f325e)

Solutions

1. Restore the side-car from a known-good backup and reload.
2. Re-index the source documents into a fresh store instead of trusting the corrupt side-car.
3. Catch the ValueError in load tooling to quarantine corrupt stores for inspection.

When it happens

Trigger: Thrown at turbovec-python/python/turbovec/haystack.py:920 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of RyanCodrai/turbovec@ccab9f325e (2026-09-06). Data as JSON: /api/errors/52b9905cef56ca84. Report an issue: GitHub.