{"record":{"id":"340b775120e7faa5","repo":"RyanCodrai/turbovec","slug":"persisted-store-is-corrupt-the-handle-watermark-n","errorCode":null,"errorMessage":"persisted store is corrupt: the handle watermark next_u64={int(next_u64)} is below the largest {what} handle in use ({max(handle_list)}). Loading it would reissue live handles on the next write.","messagePattern":"persisted store is corrupt: the handle watermark next_u64=(.+?) is below the largest (.+?) handle in use \\((.+?)\\)\\. Loading it would reissue live handles on the next write\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"turbovec-python/python/turbovec/_persist.py","lineNumber":471,"sourceCode":"    if len(set(handle_list)) != len(handle_list):\n        raise ValueError(\n            f\"persisted store is corrupt: duplicate {what} handles in the side-car\"\n        )\n    if len(handle_list) != n_index:\n        raise ValueError(\n            f\"persisted store is inconsistent with its index: side-car has \"\n            f\"{len(handle_list)} {what} handle(s) but the index holds {n_index}. \"\n            f\"The .tvim index and its JSON side-car are out of sync.\"\n        )\n    for h in handle_list:\n        if not index.contains(h):\n            raise ValueError(\n                f\"persisted store is inconsistent with its index: a {what} in \"\n                f\"the side-car has no vector in the index (internal record id \"\n                f\"{h}). The .tvim index and its JSON side-car are out of sync.\"\n            )\n    if next_u64 is not None and handle_list and int(next_u64) < max(handle_list):\n        raise ValueError(\n            f\"persisted store is corrupt: the handle watermark next_u64=\"\n            f\"{int(next_u64)} is below the largest {what} handle in use \"\n            f\"({max(handle_list)}). Loading it would reissue live handles \"\n            f\"on the next write.\"\n        )\n\n\ndef check_sidecar_keysets(\n    mapping_keys: Iterable,\n    sidecar_keys: Iterable,\n    *,\n    what: str = \"entry\",\n    mapping_name: str = \"id map\",\n    sidecar_name: str = \"payload map\",\n) -> None:\n    \"\"\"Validate that two side-car structures keyed by the same ids agree.\n\n    Args:","sourceCodeStart":453,"sourceCodeEnd":489,"githubUrl":"https://github.com/RyanCodrai/turbovec/blob/ccab9f325e6ce2a270a87daf01ae4e443bcf2d49/turbovec-python/python/turbovec/_persist.py#L453-L489","documentation":"The side-car stores next_u64, the handle-allocator watermark: the next handle the store will hand out. If it is not strictly greater than every handle currently in use, loading the store would reissue live handles on the next write, aliasing new records onto existing ones. check_persisted_handles rejects such a pair as corrupt.","triggerScenarios":"load / load_from_disk / from_persist_path on a pair where next_u64 <= max(handles): e.g. someone edited the watermark down, an old side-car was paired with a newer index, or handles were appended without bumping the watermark.","commonSituations":"Hand-editing the JSON side-car and resetting next_u64 to 0 or the record count; merging side-cars from two exports keeping the smaller watermark; a buggy external tool rewriting the field.","solutions":["Restore the pair from a consistent backup; do not just raise the watermark unless you are certain no other corruption exists.","If repairing deliberately, set next_u64 to max(handles) + 1 and verify counts/containment checks also pass.","Prefer rebuilding the store from source data over manual watermark surgery.","Pre-check before load: `next_u64 is None or next_u64 > max(handles)` and abort with a clear message otherwise."],"exampleFix":"// before\n# \"next_u64\": 3, \"handles\": [3, 5]  -> watermark rewound\n// after\n# \"next_u64\": 6, \"handles\": [3, 5]  (or restore from backup)","handlingStrategy":"validation","validationCode":"def watermark_sound(next_u64, handles) -> bool:\n    hs = [int(h) for h in handles]\n    return next_u64 is None or not hs or int(next_u64) > max(hs)\n# check the side-car before load","typeGuard":null,"tryCatchPattern":"try:\n    store = turbovec.load(path)\nexcept ValueError as e:\n    if 'watermark' in str(e):\n        restore_from_backup()  # or repair next_u64 = max(handles)+1, then revalidate\n    else:\n        raise","preventionTips":["Never hand-edit next_u64 in the side-car.","When merging/restoring side-cars, keep the largest watermark, never the smallest.","After any manual repair, re-run all handle checks (duplicates, count, containment, watermark).","Back up the pair immediately after each successful save so repairs are rare."],"tags":["python","persistence","corruption","handle-allocation"],"backgroundTag":"checksum-mismatch","analyzedSha":"ccab9f325e6ce2a270a87daf01ae4e443bcf2d49","analyzedAt":"2026-09-06T08:39:18.516Z","contentChangedAt":"2026-09-06T08:39:18.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}