MemPalace/mempalace · error · ValueError

metadata key {key!r} clashes with a reserved Milvus field

Error message

metadata key {key!r} clashes with a reserved Milvus field

What it means

Raised by _jsonable_metadata() when a metadata dict uses one of the reserved field names: id, document, metadata, vector, sparse, distance, score. These are the Milvus collection's structural columns, so metadata keys cannot shadow them; the backend raises ValueError before insert rather than letting the payload collide. Values that are not JSON-serializable are stringified instead (no error) — only key collisions raise.

Source

Thrown at mempalace/backends/milvus.py:241

    if len(dims) > 1:
        raise DimensionMismatchError(f"milvus batch cannot mix embedding dimensions {sorted(dims)}")
    return vectors, dims.pop() if dims else 0


def _clean_text(value: Any) -> str:
    text = "" if value is None else str(value)
    return strip_lone_surrogates(text).replace("\x00", "")


def _utf8_len(value: str) -> int:
    return len(value.encode("utf-8"))


def _jsonable_metadata(meta: dict | None) -> dict:
    cleaned = {}
    for key, value in (meta or {}).items():
        if key in RESERVED_FIELDS:
            raise ValueError(f"metadata key {key!r} clashes with a reserved Milvus field")
        try:
            json.dumps(value, ensure_ascii=False)
        except (TypeError, ValueError):
            value = str(value)
        cleaned[str(key)] = value
    return cleaned


def _slug(value: str, fallback: str = "collection") -> str:
    safe = re.sub(r"[^A-Za-z0-9_]+", "_", value).strip("_")
    if not safe or not re.match(r"^[A-Za-z_]", safe):
        safe = f"{fallback}_{safe}" if safe else fallback
    if len(safe) <= 120:
        return safe
    digest = sha256(value.encode("utf-8", errors="surrogatepass")).hexdigest()[:12]
    return f"{safe[:107]}_{digest}"

View on GitHub (pinned to 06cb6987f0)

Solutions

1. Prefix or rename reserved keys at ingest: source_id, source_document, relevance_score
2. Add a metadata sanitizer that maps RESERVED_FIELDS names before calling add
3. If the collision is intentional (e.g. drawer id), use the backend's dedicated id parameter instead of metadata

Example fix

# before
collection.add(ids=[i], documents=[d], metadatas=[{"id": i, "score": 0.9}])

# after
collection.add(ids=[i], documents=[d], metadatas=[{"source_id": i, "relevance_score": 0.9}])

RESERVED = {"id", "document", "metadata", "vector", "sparse", "distance", "score"}

def sanitize_metadata(meta: dict) -> dict:
    return {(f"src_{k}" if k in RESERVED else k): v for k, v in (meta or {}).items()}

try:
    collection.add(ids=ids, documents=docs, metadatas=metas)
except ValueError as e:
    if "reserved Milvus field" in str(e):
        metas = [sanitize_metadata(m) for m in metas]
        collection.add(ids=ids, documents=docs, metadatas=metas)
    else:
        raise

When it happens

Trigger: add(..., metadatas=[{"id": "x", "document": "text", "score": 1.5}]) — any of the seven reserved names used as a metadata key.

Common situations: Ingesting raw records that carry their own 'id', 'document', or 'score' fields; porting ChromaDB metadata that happened to use reserved words.

Related errors

• Milvus filter field {name!r} is not a safe identifier
• $in requires a non-empty list for {field!r}
• $nin requires a non-empty list for {field!r}
• embedding must be a non-empty 1D vector
• embedding dimension must be positive

AI-assisted analysis of MemPalace/mempalace@06cb6987f0 (2026-08-15). Data as JSON: /api/errors/d2e0bf8173d7e7e5. Report an issue: GitHub.