{"record":{"id":"52b9905cef56ca84","repo":"RyanCodrai/turbovec","slug":"persisted-store-is-corrupt-duplicate-document-ids","errorCode":null,"errorMessage":"persisted store is corrupt: duplicate document ids in the side-car","messagePattern":"persisted store is corrupt: duplicate document ids in the side-car","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"turbovec-python/python/turbovec/haystack.py","lineNumber":920,"sourceCode":"        store._index = IdMapIndex.load(str(folder / \"index.tvim\"))\n        # Reconstruct {int handle: doc data} from the list-of-pairs form.\n        # `_deserialize_doc_data` is shape-tolerant: v1 entries lack the\n        # `blob` / `sparse_embedding` keys and come back with both set to\n        # None, which matches their original on-write state.\n        store._u64_to_doc = {\n            int(h): cls._deserialize_doc_data(d) for h, d in state[\"u64_to_doc\"]\n        }\n        store._next_u64 = state[\"next_u64\"]\n        # Rebuild str_to_u64 from the reloaded doc table.\n        store._str_to_u64 = {\n            data[\"id\"]: handle for handle, data in store._u64_to_doc.items()\n        }\n        # Two handles sharing a document id would silently collapse in the\n        # rebuild above, leaving a shadow document that is searchable but\n        # unreachable (and undeletable) by id. The write path enforces\n        # unique ids, so a duplicate can only mean a corrupt side-car.\n        if len(store._str_to_u64) != len(store._u64_to_doc):\n            raise ValueError(\n                \"persisted store is corrupt: duplicate document ids in the side-car\"\n            )\n        check_persisted_handles(\n            store._index,\n            store._u64_to_doc.keys(),\n            what=\"document\",\n            next_u64=store._next_u64,\n        )\n        return store\n\n    # ---- Copy & pickle ------------------------------------------------\n    #\n    # The Rust index is not directly picklable; it round-trips through\n    # the core's in-memory ``.tvim`` byte format\n    # (``IdMapIndex.to_bytes`` / ``from_bytes``). The per-store lock and\n    # the async executor are excluded from the state — neither can cross\n    # pickling — and recreated on restore; a restored/copied store always\n    # owns a fresh executor, even when the original wrapped a","sourceCodeStart":902,"sourceCodeEnd":938,"githubUrl":"https://github.com/RyanCodrai/turbovec/blob/ccab9f325e6ce2a270a87daf01ae4e443bcf2d49/turbovec-python/python/turbovec/haystack.py#L902-L938","documentation":"Raised in load_from_disk after rebuilding _str_to_u64 from the reloaded doc table: the rebuilt map is shorter than the handle table, meaning two handles shared one document id. The write path enforces unique ids, so duplicates can only come from a hand-edited or otherwise corrupt docstore.json side-car; left unchecked, a duplicate would silently collapse into a shadow document that is searchable but unreachable and undeletable by id.","triggerScenarios":"Thrown at turbovec-python/python/turbovec/haystack.py:920 when the library encounters an invalid state.","commonSituations":"See trigger scenarios.","solutions":["Restore the side-car from a known-good backup and reload.","Re-index the source documents into a fresh store instead of trusting the corrupt side-car.","Catch the ValueError in load tooling to quarantine corrupt stores for inspection."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"ccab9f325e6ce2a270a87daf01ae4e443bcf2d49","analyzedAt":"2026-09-06T08:39:18.516Z","contentChangedAt":"2026-09-06T08:39:18.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}