{"record":{"id":"6d74037e32bc001f","repo":"MemPalace/mempalace","slug":"milvus-batch-cannot-mix-embedding-dimensions-sort","errorCode":null,"errorMessage":"milvus batch cannot mix embedding dimensions {sorted(dims)}","messagePattern":"milvus batch cannot mix embedding dimensions (.+?)","errorType":"exception","errorClass":"DimensionMismatchError","httpStatus":null,"severity":"error","filePath":"mempalace/backends/milvus.py","lineNumber":224,"sourceCode":"    return \"(\" + \") and (\".join(present) + \")\"\n\n\ndef _as_vector_array(vector: list[float]) -> np.ndarray:\n    arr = np.asarray(vector, dtype=np.float32)\n    if arr.ndim != 1 or arr.size == 0:\n        raise ValueError(\"embedding must be a non-empty 1D vector\")\n    return arr\n\n\ndef _normalize_vectors(embeddings: list[list[float]]) -> tuple[list[list[float]], int]:\n    vectors = []\n    dims = set()\n    for embedding in embeddings:\n        arr = _as_vector_array(embedding)\n        vectors.append(arr.astype(float).tolist())\n        dims.add(int(arr.size))\n    if len(dims) > 1:\n        raise DimensionMismatchError(f\"milvus batch cannot mix embedding dimensions {sorted(dims)}\")\n    return vectors, dims.pop() if dims else 0\n\n\ndef _clean_text(value: Any) -> str:\n    text = \"\" if value is None else str(value)\n    return strip_lone_surrogates(text).replace(\"\\x00\", \"\")\n\n\ndef _utf8_len(value: str) -> int:\n    return len(value.encode(\"utf-8\"))\n\n\ndef _jsonable_metadata(meta: dict | None) -> dict:\n    cleaned = {}\n    for key, value in (meta or {}).items():\n        if key in RESERVED_FIELDS:\n            raise ValueError(f\"metadata key {key!r} clashes with a reserved Milvus field\")\n        try:","sourceCodeStart":206,"sourceCodeEnd":242,"githubUrl":"https://github.com/MemPalace/mempalace/blob/06cb6987f02610784fefbad4b2bd5d026d164ba6/mempalace/backends/milvus.py#L206-L242","documentation":"Raised by _normalize_vectors() when a single add/upsert batch contains embeddings of differing dimensions. Milvus collections are fixed-dimension, so a mixed batch cannot be inserted; the backend collects all dimensions first and raises DimensionMismatchError listing them. Distinct from collection-vs-batch mismatch (a different error raised later with the collection's expected dim) — this is strictly batch-internal inconsistency.","triggerScenarios":"add(embeddings=[[0.1]*384, [0.1]*768]) — one embedder call returned 384-d and another 768-d vectors concatenated into one batch.","commonSituations":"Switching embedding models mid-stream; mixing cached embeddings from an old model with fresh ones; multiple embedders (dense vs sparse) merged accidentally.","solutions":["Verify all embeddings in a batch come from the same model/dimension before calling add","Re-embed stale data after changing models, or clear the collection first","Group by dimension and insert as separate batches/collections"],"exampleFix":"# before\ncollection.add(ids=ids, documents=docs, embeddings=all_embs)  # mixed 384/768\n\n# after\ndims = {len(e) for e in all_embs}\nif len(dims) > 1:\n    raise ValueError(f\"mixed embedding dimensions: {dims}\")\ncollection.add(ids=ids, documents=docs, embeddings=all_embs)","handlingStrategy":"validation","validationCode":"def uniform_dimension(embeddings) -> int | None:\n    dims = {len(e) for e in embeddings}\n    if len(dims) != 1:\n        raise ValueError(f\"mixed embedding dimensions: {sorted(dims)}\")\n    return dims.pop() if dims else None","typeGuard":null,"tryCatchPattern":"from mempalace.backends.base import DimensionMismatchError\ntry:\n    collection.add(ids=ids, documents=docs, embeddings=embs)\nexcept DimensionMismatchError as e:\n    if \"cannot mix\" in str(e):\n        groups = {}\n        for i, d, e_ in zip(ids, docs, embs):\n            groups.setdefault(len(e_), []).append((i, d, e_))\n        for batch in groups.values():\n            collection.add(ids=[b[0] for b in batch], documents=[b[1] for b in batch], embeddings=[b[2] for b in batch])\n    else:\n        raise","preventionTips":["Pin one embedding model per collection and record its dimension in config","Re-embed everything after model switches instead of mixing","Validate dimension uniformity at the top of every ingest pipeline"],"tags":["milvus","embeddings","dimension-mismatch","batch"],"backgroundTag":null,"analyzedSha":"06cb6987f02610784fefbad4b2bd5d026d164ba6","analyzedAt":"2026-08-15T03:03:36.213Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}