pathwaycom/pathway · error · ValueError

Milvus collection {collection_name!r} does not exist; create

Error message

Milvus collection {collection_name!r} does not exist; create it before writing. pw.io.milvus.write never creates a collection because it cannot infer the vector field's dimension.

What it means

pw.io.milvus.write never creates collections, because it cannot infer the vector field's dimension from a Pathway table alone. Before writing, it checks client.has_collection(collection_name) and, if the collection is missing, closes the client and raises ValueError telling you to create it first. This fails fast instead of erroring deep inside pw.run() — or, for an empty table, never at all.

Source

Thrown at python/pathway/io/milvus/__init__.py:305

    if batch_size < 1:
        raise ValueError(f"batch_size must be a positive integer, got {batch_size}.")

    if primary_key._table is not table:
        raise ValueError(
            f"primary_key column {primary_key._name!r} does not belong to the "
            f"provided table. Pass a column reference from the same table, "
            f"e.g. primary_key=table.{primary_key._name}."
        )

    client = _make_client(MilvusClient, uri)

    # Fail fast if the collection is missing: otherwise the error would only
    # surface deep inside pw.run() on the first upsert, or — for an empty table —
    # never, silently running a misconfigured pipeline that writes nothing.
    if not client.has_collection(collection_name):
        client.close()
        raise ValueError(
            f"Milvus collection {collection_name!r} does not exist; create it "
            f"before writing. pw.io.milvus.write never creates a collection "
            f"because it cannot infer the vector field's dimension."
        )

    pk = primary_key._name
    # Accumulates (is_addition, row) in arrival order for the current batch.
    _buffer: list[tuple] = []

    def on_change(key, row, time, is_addition):
        _buffer.append((is_addition, _prepare_row(row)))

    def on_time_end(time):
        to_delete = []
        to_upsert = []

        for is_add, row in _buffer:
            if is_add:

View on GitHub (pinned to fa2f74a464)

Solutions

Create the collection up front with pymilvus MilvusClient.create_schema/create_collection, defining the vector field with the correct dimension matching your embeddings
Verify the name: print(client.list_collections()) and fix the collection_name typo
Keep a small bootstrap script that ensures the schema exists and run it before pipeline startup / deployment

Example fix

# before
pw.io.milvus.write(t, uri="./milvus.db", collection_name="docs", primary_key=t.id)  # 'docs' missing

# after
from pymilvus import MilvusClient, DataType
client = MilvusClient("./milvus.db")
if not client.has_collection("docs"):
    schema = client.create_schema(auto_id=False)
    schema.add_field("id", DataType.INT64, is_primary=True)
    schema.add_field("vector", DataType.FLOAT_VECTOR, dim=384)
    client.create_collection("docs", schema)
pw.io.milvus.write(t, uri="./milvus.db", collection_name="docs", primary_key=t.id)

Defensive patterns

Strategy: validation

Validate before calling

from pymilvus import MilvusClient

client = MilvusClient(uri)
if not client.has_collection(collection_name):
    raise ValueError(f"Create collection {collection_name!r} (with vector dim) before pw.io.milvus.write")
client.close()

Try / catch

try:
    pw.io.milvus.write(table, uri, collection_name, primary_key=table.id)
except ValueError as e:
    if "does not exist" in str(e):
        ensure_collection(uri, collection_name, dim=len(table.emb[0]))  # your bootstrap
    else:
        raise

Prevention

Run a bootstrap script that creates the collection (with explicit vector dimension) before the pipeline
Check client.list_collections() when writes target a shared server
Drive collection_name from one shared constant to avoid environment drift

When it happens

Trigger: Calling pw.io.milvus.write(table, uri, collection_name='docs', ...) before any collection named 'docs' exists on the server; also after dropping the collection or pointing collection_name at a typo'd name.

Common situations: First run of a new pipeline against a fresh Milvus instance; typos in collection_name; environment mismatch (writing to a dev URI while the collection was created in another environment).

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/bb117ee32905286d. Report an issue: GitHub.