RyanCodrai/turbovec · critical · ValueError

persisted store is corrupt: the handle watermark next_u64={i

Error message

persisted store is corrupt: the handle watermark next_u64={int(next_u64)} is below the largest {what} handle in use ({max(handle_list)}). Loading it would reissue live handles on the next write.

What it means

The side-car stores next_u64, the handle-allocator watermark: the next handle the store will hand out. If it is not strictly greater than every handle currently in use, loading the store would reissue live handles on the next write, aliasing new records onto existing ones. check_persisted_handles rejects such a pair as corrupt.

Source

Thrown at turbovec-python/python/turbovec/_persist.py:471

    if len(set(handle_list)) != len(handle_list):
        raise ValueError(
            f"persisted store is corrupt: duplicate {what} handles in the side-car"
        )
    if len(handle_list) != n_index:
        raise ValueError(
            f"persisted store is inconsistent with its index: side-car has "
            f"{len(handle_list)} {what} handle(s) but the index holds {n_index}. "
            f"The .tvim index and its JSON side-car are out of sync."
        )
    for h in handle_list:
        if not index.contains(h):
            raise ValueError(
                f"persisted store is inconsistent with its index: a {what} in "
                f"the side-car has no vector in the index (internal record id "
                f"{h}). The .tvim index and its JSON side-car are out of sync."
            )
    if next_u64 is not None and handle_list and int(next_u64) < max(handle_list):
        raise ValueError(
            f"persisted store is corrupt: the handle watermark next_u64="
            f"{int(next_u64)} is below the largest {what} handle in use "
            f"({max(handle_list)}). Loading it would reissue live handles "
            f"on the next write."
        )


def check_sidecar_keysets(
    mapping_keys: Iterable,
    sidecar_keys: Iterable,
    *,
    what: str = "entry",
    mapping_name: str = "id map",
    sidecar_name: str = "payload map",
) -> None:
    """Validate that two side-car structures keyed by the same ids agree.

    Args:

View on GitHub (pinned to ccab9f325e)

Solutions

Restore the pair from a consistent backup; do not just raise the watermark unless you are certain no other corruption exists.
If repairing deliberately, set next_u64 to max(handles) + 1 and verify counts/containment checks also pass.
Prefer rebuilding the store from source data over manual watermark surgery.
Pre-check before load: `next_u64 is None or next_u64 > max(handles)` and abort with a clear message otherwise.

Example fix

// before
# "next_u64": 3, "handles": [3, 5]  -> watermark rewound
// after
# "next_u64": 6, "handles": [3, 5]  (or restore from backup)

Defensive patterns

Strategy: validation

Validate before calling

def watermark_sound(next_u64, handles) -> bool:
    hs = [int(h) for h in handles]
    return next_u64 is None or not hs or int(next_u64) > max(hs)
# check the side-car before load

Try / catch

try:
    store = turbovec.load(path)
except ValueError as e:
    if 'watermark' in str(e):
        restore_from_backup()  # or repair next_u64 = max(handles)+1, then revalidate
    else:
        raise

Prevention

Never hand-edit next_u64 in the side-car.
When merging/restoring side-cars, keep the largest watermark, never the smallest.
After any manual repair, re-run all handle checks (duplicates, count, containment, watermark).
Back up the pair immediately after each successful save so repairs are rare.

When it happens

Trigger: load / load_from_disk / from_persist_path on a pair where next_u64 <= max(handles): e.g. someone edited the watermark down, an old side-car was paired with a newer index, or handles were appended without bumping the watermark.

Common situations: Hand-editing the JSON side-car and resetting next_u64 to 0 or the record count; merging side-cars from two exports keeping the smaller watermark; a buggy external tool rewriting the field.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of RyanCodrai/turbovec@ccab9f325e (2026-09-06). Data as JSON: /api/errors/340b775120e7faa5. Report an issue: GitHub.