{"record":{"id":"bcf811ebfbc6832f","repo":"MemPalace/mempalace","slug":"pre-mining-collision-scan-detected-len-collisions","errorCode":null,"errorMessage":"Pre-mining collision scan detected {len(collisions)} colliding drawer_id{'s' if len(collisions) != 1 else ''}: ... Each colliding drawer_id would cause the second ChromaDB upsert to silently overwrite the first. Fix the upstream chunker / miner to emit distinct keys, or investigate the SHA-256 hash collision.","messagePattern":"Pre-mining collision scan detected (.+?) colliding drawer_id(.+?): \\.\\.\\. Each colliding drawer_id would cause the second ChromaDB upsert to silently overwrite the first\\. Fix the upstream chunker / miner to emit distinct keys, or investigate the SHA-256 hash collision\\.","errorType":"exception","errorClass":"CollisionError","httpStatus":null,"severity":"critical","filePath":"mempalace/collision_scan.py","lineNumber":99,"sourceCode":"        incoming[drawer_id].add(_metadata_key(meta))\n\n    # Query existing rows for any incoming id. ChromaDB's get(ids=...)\n    # returns only the rows whose ids are present; missing ids are\n    # silently absent from the result, which is what we want.\n    incoming_ids = list(incoming.keys())\n    result = collection.get(ids=incoming_ids, include=[\"metadatas\"])\n    existing_ids: list = result[\"ids\"] if hasattr(result, \"__getitem__\") else []\n    existing_metas: list = result[\"metadatas\"] if existing_ids else []\n\n    # Merge existing metadata into the incoming map. A real collision is\n    # a drawer_id whose incoming + existing metadata key tuples are not\n    # all the same.\n    for drawer_id, meta in zip(existing_ids, existing_metas):\n        incoming[drawer_id].add(_metadata_key(meta or {}))\n\n    collisions = {did: keys for did, keys in incoming.items() if len(keys) > 1}\n    if collisions:\n        raise CollisionError(_format_collisions(collisions))\n\n\ndef _format_collisions(collisions: dict[str, set[tuple]]) -> str:\n    \"\"\"Render a CollisionError message that enumerates every colliding\n    drawer_id and the metadata tuples producing it.\"\"\"\n    lines = [\n        f\"Pre-mining collision scan detected {len(collisions)} \"\n        f\"colliding drawer_id{'s' if len(collisions) != 1 else ''}:\",\n    ]\n    for drawer_id, keys in sorted(collisions.items()):\n        lines.append(f\"  {drawer_id}:\")\n        for key in sorted(keys, key=lambda k: tuple(str(part) for part in k)):\n            if len(key) == 1:\n                lines.append(f\"    source_file={key[0]!r}\")\n            else:\n                lines.append(f\"    source_file={key[0]!r}, chunk_index={key[1]!r}\")\n    lines.append(\n        \"Each colliding drawer_id would cause the second ChromaDB upsert \"","sourceCodeStart":81,"sourceCodeEnd":117,"githubUrl":"https://github.com/MemPalace/mempalace/blob/06cb6987f02610784fefbad4b2bd5d026d164ba6/mempalace/collision_scan.py#L81-L117","documentation":"Raised by the pre-mining collision scan when two or more incoming chunks share the same drawer_id but carry different metadata key tuples (or collide with an existing stored record with different metadata). ChromaDB upserts are keyed by id, so the second write would silently overwrite the first — this scan converts that silent data loss into a hard CollisionError before any write happens. Only a genuine SHA-256 collision or a chunker/miner emitting duplicate keys produces it; identical metadata for the same id is treated as a benign re-upsert.","triggerScenarios":"Mining a file set where two distinct chunks hash to the same drawer_id (broken/deterministic-degenerate chunker, e.g. chunks differing only in fields excluded from the hash), or re-mining content where the metadata generation changed between runs so the same id now carries a different metadata tuple.","commonSituations":"Upgrading the miner or metadata schema so previously-stable drawer_ids now get different metadata; a custom chunker that drops the differentiating field before hashing; duplicate source files with near-identical content fed in one batch; a genuine (astronomically unlikely) SHA-256 collision.","solutions":["Inspect the emitted collision report: it lists each colliding drawer_id and the metadata tuples producing it — identify which field differs","Fix the upstream chunker/miner so the differing field is included in the drawer_id hash (or ids are otherwise made unique)","If the metadata schema changed intentionally, migrate or clear the affected collection records so old and new tuples do not mix","Verify each colliding chunk is actually distinct content; if it is true duplicate content, deduplicate before mining"],"exampleFix":"# before (chunker hashes only content, ignoring source path)\ndrawer_id = sha256(chunk.text)\n\n# after (include source-identifying metadata so identical text from\ndifferent sources gets distinct ids)\ndrawer_id = sha256(source_path + '\\n' + chunk.text)","handlingStrategy":"validation","validationCode":"from collections import defaultdict\n\n# Before mining: group incoming records by drawer_id\nby_id = defaultdict(set)\nfor rec in records:\n    by_id[rec.drawer_id].add(tuple(sorted(rec.metadata.items())))\ncollisions = {k: v for k, v in by_id.items() if len(v) > 1}\nassert not collisions, f\"fix chunker: colliding ids {sorted(collisions)}\"","typeGuard":null,"tryCatchPattern":"try:\n    run_collision_scan(collection, records)\nexcept CollisionError as exc:\n    # do NOT catch-and-continue: this guard prevents silent ChromaDB overwrites\n    log.error(\"chunker emitted duplicate drawer_ids: %s\", exc)\n    raise SystemExit(2)","preventionTips":["Never suppress CollisionError — it is the only thing standing between you and silent data loss","Include every metadata field that differentiates chunks in the drawer_id hash input","After changing miner/metadata schemas, run a small re-mine and diff drawer_id->metadata mappings before full ingest"],"tags":["data-integrity","chromadb","mining","hashing","mempalace"],"backgroundTag":null,"analyzedSha":"06cb6987f02610784fefbad4b2bd5d026d164ba6","analyzedAt":"2026-08-15T03:03:36.213Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}