RyanCodrai/turbovec · error · ValueError

persisted store is corrupt: duplicate {what} handles in the

Error message

persisted store is corrupt: duplicate {what} handles in the side-car

What it means

check_persisted_handles verifies that the handle list stored in the JSON side-car is sound relative to the .tvim vector index before loading. If the same handle appears twice the side-car is corrupt, and loading would map two records onto one handle, so it raises ValueError immediately.

Source

Thrown at turbovec-python/python/turbovec/_persist.py:454

    Args:
        index: the loaded ``IdMapIndex`` (uses ``len`` and ``contains``).
        handles: the u64 handles the side-car maps can resolve.
        what: noun for error messages (e.g. "document", "node").
        next_u64: the side-car's handle watermark, if the caller has it.
            Handles are issued by pre-incrementing it, so it must be at
            least the largest handle in use; a smaller value reissues live
            handles on the next write (issue #321).

    Raises:
        ValueError: if the side-car has duplicate handles, a different count
            than the index, a handle the index doesn't contain, or a
            watermark below the largest handle in use.
    """
    handle_list = [int(h) for h in handles]
    n_index = len(index)

    if len(set(handle_list)) != len(handle_list):
        raise ValueError(
            f"persisted store is corrupt: duplicate {what} handles in the side-car"
        )
    if len(handle_list) != n_index:
        raise ValueError(
            f"persisted store is inconsistent with its index: side-car has "
            f"{len(handle_list)} {what} handle(s) but the index holds {n_index}. "
            f"The .tvim index and its JSON side-car are out of sync."
        )
    for h in handle_list:
        if not index.contains(h):
            raise ValueError(
                f"persisted store is inconsistent with its index: a {what} in "
                f"the side-car has no vector in the index (internal record id "
                f"{h}). The .tvim index and its JSON side-car are out of sync."
            )
    if next_u64 is not None and handle_list and int(next_u64) < max(handle_list):
        raise ValueError(
            f"persisted store is corrupt: the handle watermark next_u64="

View on GitHub (pinned to ccab9f325e)

Solutions

  1. Regenerate the store from source data (re-add the documents) rather than repairing the side-car by hand.
  2. If repairing manually, remove duplicate handle entries and ensure the count matches the index (run the other checks: count and containment).
  3. Restore the .tvim + JSON pair from backup.
  4. Validate before load: `len(handles) == len(set(handles))` wrapped in try/except ValueError for a clean error message.

Example fix

// before
# side-car: "handles": [3, 5, 3]  -> duplicate
// after
# side-car: "handles": [3, 5]  (or restore from backup / rebuild store)
Defensive patterns

Strategy: validation

Validate before calling

def handles_unique(handles) -> bool:
    hs = [int(h) for h in handles]
    return len(hs) == len(set(hs))
# run on the side-car's handle arrays before load

Try / catch

try:
    store = turbovec.load(path)
except ValueError as e:
    if 'duplicate' in str(e) and 'handles' in str(e):
        restore_from_backup()  # side-car is corrupt
    else:
        raise

Prevention

When it happens

Trigger: Calling load / load_from_disk / from_persist_path / _load_from (or check_persisted_handles directly) on a persisted pair whose docs- or metadata-handle array in the side-car contains a repeated integer, usually from manual editing, a botched merge of two files, or partial overwrite.

Common situations: Hand-editing or scripting the JSON side-car; concatenating/appending entries from two exports; a crashed external tool that duplicated lines; file-sync tools merging conflicting versions.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of RyanCodrai/turbovec@ccab9f325e (2026-09-06). Data as JSON: /api/errors/f7008d43c154b189. Report an issue: GitHub.