MemPalace/mempalace · critical · CollisionError

Pre-mining collision scan detected {len(collisions)} collidi

Error message

Pre-mining collision scan detected {len(collisions)} colliding drawer_id{'s' if len(collisions) != 1 else ''}: ... Each colliding drawer_id would cause the second ChromaDB upsert to silently overwrite the first. Fix the upstream chunker / miner to emit distinct keys, or investigate the SHA-256 hash collision.

What it means

Raised by the pre-mining collision scan when two or more incoming chunks share the same drawer_id but carry different metadata key tuples (or collide with an existing stored record with different metadata). ChromaDB upserts are keyed by id, so the second write would silently overwrite the first — this scan converts that silent data loss into a hard CollisionError before any write happens. Only a genuine SHA-256 collision or a chunker/miner emitting duplicate keys produces it; identical metadata for the same id is treated as a benign re-upsert.

Source

Thrown at mempalace/collision_scan.py:99

        incoming[drawer_id].add(_metadata_key(meta))

    # Query existing rows for any incoming id. ChromaDB's get(ids=...)
    # returns only the rows whose ids are present; missing ids are
    # silently absent from the result, which is what we want.
    incoming_ids = list(incoming.keys())
    result = collection.get(ids=incoming_ids, include=["metadatas"])
    existing_ids: list = result["ids"] if hasattr(result, "__getitem__") else []
    existing_metas: list = result["metadatas"] if existing_ids else []

    # Merge existing metadata into the incoming map. A real collision is
    # a drawer_id whose incoming + existing metadata key tuples are not
    # all the same.
    for drawer_id, meta in zip(existing_ids, existing_metas):
        incoming[drawer_id].add(_metadata_key(meta or {}))

    collisions = {did: keys for did, keys in incoming.items() if len(keys) > 1}
    if collisions:
        raise CollisionError(_format_collisions(collisions))


def _format_collisions(collisions: dict[str, set[tuple]]) -> str:
    """Render a CollisionError message that enumerates every colliding
    drawer_id and the metadata tuples producing it."""
    lines = [
        f"Pre-mining collision scan detected {len(collisions)} "
        f"colliding drawer_id{'s' if len(collisions) != 1 else ''}:",
    ]
    for drawer_id, keys in sorted(collisions.items()):
        lines.append(f"  {drawer_id}:")
        for key in sorted(keys, key=lambda k: tuple(str(part) for part in k)):
            if len(key) == 1:
                lines.append(f"    source_file={key[0]!r}")
            else:
                lines.append(f"    source_file={key[0]!r}, chunk_index={key[1]!r}")
    lines.append(
        "Each colliding drawer_id would cause the second ChromaDB upsert "

View on GitHub (pinned to 06cb6987f0)

Solutions

Inspect the emitted collision report: it lists each colliding drawer_id and the metadata tuples producing it — identify which field differs
Fix the upstream chunker/miner so the differing field is included in the drawer_id hash (or ids are otherwise made unique)
If the metadata schema changed intentionally, migrate or clear the affected collection records so old and new tuples do not mix
Verify each colliding chunk is actually distinct content; if it is true duplicate content, deduplicate before mining

Example fix

# before (chunker hashes only content, ignoring source path)
drawer_id = sha256(chunk.text)

# after (include source-identifying metadata so identical text from
different sources gets distinct ids)
drawer_id = sha256(source_path + '\n' + chunk.text)

Defensive patterns

Strategy: validation

Validate before calling

from collections import defaultdict

# Before mining: group incoming records by drawer_id
by_id = defaultdict(set)
for rec in records:
    by_id[rec.drawer_id].add(tuple(sorted(rec.metadata.items())))
collisions = {k: v for k, v in by_id.items() if len(v) > 1}
assert not collisions, f"fix chunker: colliding ids {sorted(collisions)}"

Try / catch

try:
    run_collision_scan(collection, records)
except CollisionError as exc:
    # do NOT catch-and-continue: this guard prevents silent ChromaDB overwrites
    log.error("chunker emitted duplicate drawer_ids: %s", exc)
    raise SystemExit(2)

Prevention

Never suppress CollisionError — it is the only thing standing between you and silent data loss
Include every metadata field that differentiates chunks in the drawer_id hash input
After changing miner/metadata schemas, run a small re-mine and diff drawer_id->metadata mappings before full ingest

When it happens

Trigger: Mining a file set where two distinct chunks hash to the same drawer_id (broken/deterministic-degenerate chunker, e.g. chunks differing only in fields excluded from the hash), or re-mining content where the metadata generation changed between runs so the same id now carries a different metadata tuple.

Common situations: Upgrading the miner or metadata schema so previously-stable drawer_ids now get different metadata; a custom chunker that drops the differentiating field before hashing; duplicate source files with near-identical content fed in one batch; a genuine (astronomically unlikely) SHA-256 collision.

Related errors

AI-assisted analysis of MemPalace/mempalace@06cb6987f0 (2026-08-15). Data as JSON: /api/errors/bcf811ebfbc6832f. Report an issue: GitHub.