{"record":{"id":"659118364a3e9d79","repo":"RyanCodrai/turbovec","slug":"persisted-store-is-corrupt-duplicate-node-handles","errorCode":null,"errorMessage":"persisted store is corrupt: duplicate node handles in the side-car","messagePattern":"persisted store is corrupt: duplicate node handles in the side-car","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"turbovec-python/python/turbovec/llama_index.py","lineNumber":1091,"sourceCode":"        # v1/v2 side-cars predate the mode field: their vectors are raw,\n        # so dot_product is the mode they actually contain — loading them\n        # that way keeps scoring byte-identical to the store that wrote\n        # them. v3+ side-cars restore the recorded mode.\n        store = cls(index=index, similarity=state.get(\"similarity\", DOT_PRODUCT))\n        # v1 entries lack `node_dict` and reconstruct as narrow TextNodes;\n        # v2 entries carry it and reconstruct with full BaseNode fidelity.\n        # `_reconstruct_node` dispatches on shape, so we just load the\n        # dict as-is.\n        store._nodes = state[\"nodes\"]\n        # Reconstruct {node_id: int handle} from the list-of-pairs form.\n        store._node_id_to_u64 = {nid: int(h) for nid, h in state[\"node_id_to_u64\"]}\n        store._u64_to_node_id = {h: nid for nid, h in store._node_id_to_u64.items()}\n        store._next_u64 = int(state[\"next_u64\"])\n        # Two node ids sharing a handle would silently collapse in the\n        # inverse map built above; require the id map to be 1:1 before\n        # trusting either direction.\n        if len(store._u64_to_node_id) != len(store._node_id_to_u64):\n            raise ValueError(\n                \"persisted store is corrupt: duplicate node handles in the side-car\"\n            )\n        # The side-car holds two structures keyed by node id (`nodes` and\n        # `node_id_to_u64`); they can desync independently of the index. A\n        # `nodes` entry missing for a mapped id would otherwise surface as\n        # a KeyError deep inside a later query (issue #133).\n        check_sidecar_keysets(\n            store._node_id_to_u64.keys(),\n            store._nodes.keys(),\n            what=\"node\",\n            mapping_name=\"node_id_to_u64\",\n            sidecar_name=\"nodes\",\n        )\n        check_persisted_handles(\n            index,\n            store._u64_to_node_id.keys(),\n            what=\"node\",\n            next_u64=store._next_u64,","sourceCodeStart":1073,"sourceCodeEnd":1109,"githubUrl":"https://github.com/RyanCodrai/turbovec/blob/ccab9f325e6ce2a270a87daf01ae4e443bcf2d49/turbovec-python/python/turbovec/llama_index.py#L1073-L1109","documentation":"During from_persist_path, the persisted side-car's node-id-to-u64 handle map is rebuilt and inverted. If the inverse map is smaller than the forward map, two node ids shared one handle — the persisted store is corrupt — so this ValueError aborts the load instead of silently collapsing nodes at query time.","triggerScenarios":"Loading a persisted store whose side-car JSON contains a node_id_to_u64 mapping where at least two distinct node ids map to the same u64 handle (torn/manual edit of the side-car, or corruption from concurrent writes by an older version without snapshotting).","commonSituations":"Manually editing or truncating the side-car JSON; a crash or concurrent persist/write from an older turbovec version produced an inconsistent file; copying the index file and side-car from different generations.","solutions":["Re-persist the store from the original data (rebuild the index and call persist with the current turbovec version).","Inspect the side-car JSON's node_id_to_u64 for duplicate handle values to confirm corruption.","Restore the persisted files from a backup taken at the same write generation (index and side-car must match)."],"exampleFix":"// before (loading known-corrupt files)\nstore = TurboQuantVectorStore.from_persist_dir(\"/data/vecstore\")\n// after\nids = [h for h in json.load(open('/data/vecstore/store_sidecar.json'))['node_id_to_u64'].values()]\nassert len(ids) == len(set(ids)), 'side-car corrupt: rebuild index'\nstore = TurboQuantVectorStore.from_persist_dir(\"/data/vecstore\")","handlingStrategy":"try-catch","validationCode":"import json\nstate = json.load(open(sidecar))\nvals = list(state['node_id_to_u64'].values())\nassert len(vals) == len(set(vals)), 'corrupt side-car: rebuild index'","typeGuard":null,"tryCatchPattern":"try:\n    store = TurboQuantVectorStore.from_persist_dir(dir)\nexcept ValueError as e:\n    if 'corrupt' in str(e):\n        rebuild_index(dir)\n        store = TurboQuantVectorStore.from_persist_dir(dir)","preventionTips":["Only upgrade turbovec versions through a re-persist of the store.","Never edit the side-car JSON by hand.","Keep index and side-car files from the same write generation (copy together, atomically).","Verify side-car consistency after any crash during persist."],"tags":["python","llama-index","persist","corruption","data-integrity"],"backgroundTag":"checksum-mismatch","analyzedSha":"ccab9f325e6ce2a270a87daf01ae4e443bcf2d49","analyzedAt":"2026-09-06T08:39:18.516Z","contentChangedAt":"2026-09-06T08:39:18.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}