RyanCodrai/turbovec · error · ValueError

persisted store is corrupt: duplicate node handles in the si

Error message

persisted store is corrupt: duplicate node handles in the side-car

What it means

During from_persist_path, the persisted side-car's node-id-to-u64 handle map is rebuilt and inverted. If the inverse map is smaller than the forward map, two node ids shared one handle — the persisted store is corrupt — so this ValueError aborts the load instead of silently collapsing nodes at query time.

Source

Thrown at turbovec-python/python/turbovec/llama_index.py:1091

        # v1/v2 side-cars predate the mode field: their vectors are raw,
        # so dot_product is the mode they actually contain — loading them
        # that way keeps scoring byte-identical to the store that wrote
        # them. v3+ side-cars restore the recorded mode.
        store = cls(index=index, similarity=state.get("similarity", DOT_PRODUCT))
        # v1 entries lack `node_dict` and reconstruct as narrow TextNodes;
        # v2 entries carry it and reconstruct with full BaseNode fidelity.
        # `_reconstruct_node` dispatches on shape, so we just load the
        # dict as-is.
        store._nodes = state["nodes"]
        # Reconstruct {node_id: int handle} from the list-of-pairs form.
        store._node_id_to_u64 = {nid: int(h) for nid, h in state["node_id_to_u64"]}
        store._u64_to_node_id = {h: nid for nid, h in store._node_id_to_u64.items()}
        store._next_u64 = int(state["next_u64"])
        # Two node ids sharing a handle would silently collapse in the
        # inverse map built above; require the id map to be 1:1 before
        # trusting either direction.
        if len(store._u64_to_node_id) != len(store._node_id_to_u64):
            raise ValueError(
                "persisted store is corrupt: duplicate node handles in the side-car"
            )
        # The side-car holds two structures keyed by node id (`nodes` and
        # `node_id_to_u64`); they can desync independently of the index. A
        # `nodes` entry missing for a mapped id would otherwise surface as
        # a KeyError deep inside a later query (issue #133).
        check_sidecar_keysets(
            store._node_id_to_u64.keys(),
            store._nodes.keys(),
            what="node",
            mapping_name="node_id_to_u64",
            sidecar_name="nodes",
        )
        check_persisted_handles(
            index,
            store._u64_to_node_id.keys(),
            what="node",
            next_u64=store._next_u64,

View on GitHub (pinned to ccab9f325e)

Solutions

  1. Re-persist the store from the original data (rebuild the index and call persist with the current turbovec version).
  2. Inspect the side-car JSON's node_id_to_u64 for duplicate handle values to confirm corruption.
  3. Restore the persisted files from a backup taken at the same write generation (index and side-car must match).

Example fix

// before (loading known-corrupt files)
store = TurboQuantVectorStore.from_persist_dir("/data/vecstore")
// after
ids = [h for h in json.load(open('/data/vecstore/store_sidecar.json'))['node_id_to_u64'].values()]
assert len(ids) == len(set(ids)), 'side-car corrupt: rebuild index'
store = TurboQuantVectorStore.from_persist_dir("/data/vecstore")
Defensive patterns

Strategy: try-catch

Validate before calling

import json
state = json.load(open(sidecar))
vals = list(state['node_id_to_u64'].values())
assert len(vals) == len(set(vals)), 'corrupt side-car: rebuild index'

Try / catch

try:
    store = TurboQuantVectorStore.from_persist_dir(dir)
except ValueError as e:
    if 'corrupt' in str(e):
        rebuild_index(dir)
        store = TurboQuantVectorStore.from_persist_dir(dir)

Prevention

When it happens

Trigger: Loading a persisted store whose side-car JSON contains a node_id_to_u64 mapping where at least two distinct node ids map to the same u64 handle (torn/manual edit of the side-car, or corruption from concurrent writes by an older version without snapshotting).

Common situations: Manually editing or truncating the side-car JSON; a crash or concurrent persist/write from an older turbovec version produced an inconsistent file; copying the index file and side-car from different generations.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of RyanCodrai/turbovec@ccab9f325e (2026-09-06). Data as JSON: /api/errors/659118364a3e9d79. Report an issue: GitHub.