pathwaycom/pathway · error · ValueError

Column {k!r} contains a {v.ndim}-dimensional numpy array. pw

Error message

Column {k!r} contains a {v.ndim}-dimensional numpy array. pw.io.milvus.write only supports 1-D arrays (for FLOAT_VECTOR / BINARY_VECTOR fields).

What it means

Milvus vector fields (FLOAT_VECTOR, BINARY_VECTOR) are flat, one-dimensional vectors. When a row handed to pw.io.milvus.write contains a numpy array with more than one dimension (or zero), Pathway raises ValueError naming the column and the offending dimensionality instead of letting Milvus silently corrupt or reject the data.

Source

Thrown at python/pathway/io/milvus/__init__.py:41

_SUPPORTED_TYPES = (bool, int, float, str, dict, list, tuple, bytes, np.ndarray)


def _prepare_row(row: dict) -> dict:
    """Convert Pathway Live Data Framework-internal types to plain Python values for pymilvus.

    Unwraps ``pw.Json`` wrapper objects, converts 1-D ``numpy.ndarray`` values
    to lists, and validates that every value belongs to a type the Milvus
    connector supports.  Raises ``TypeError`` with a descriptive message for
    unsupported types, and ``ValueError`` for multi-dimensional arrays or for a
    vector containing a non-finite (NaN / infinity) component.
    """
    result = {}
    for k, v in row.items():
        if isinstance(v, _PwJson):
            v = v.value
        if isinstance(v, np.ndarray):
            if v.ndim != 1:
                raise ValueError(
                    f"Column {k!r} contains a {v.ndim}-dimensional numpy array. "
                    f"pw.io.milvus.write only supports 1-D arrays (for "
                    f"FLOAT_VECTOR / BINARY_VECTOR fields)."
                )
            v = v.tolist()
        elif not isinstance(v, _SUPPORTED_TYPES):
            raise TypeError(
                f"Column {k!r} contains a value of unsupported type "
                f"{type(v).__name__!r}. pw.io.milvus.write supports the "
                f"following Pathway types: int, float, str, bool, pw.Json, "
                f"list[float], bytes, and numpy.ndarray (1-D only)."
            )
        # A FLOAT_VECTOR (list / tuple / 1-D array of floats) with a non-finite
        # component is silently stored by Milvus and corrupts the index —
        # distances against NaN/infinity are meaningless. Reject it up front with
        # a clear, column-named error, as the other vector sinks do.
        if isinstance(v, (list, tuple)) and any(
            isinstance(x, float) and not math.isfinite(x) for x in v

View on GitHub (pinned to fa2f74a464)

Solutions

Fix the UDF to return one 1-D vector per row (index the batch: embeddings[i])
Flatten/validate in the transform step: pw.this.emb.apply(lambda a: np.asarray(a).reshape(-1))
Inspect shapes before writing: assert np.asarray(v).ndim == 1 for sampled values

Example fix

# before
@pw.udf
def embed(texts: list[str]) -> list[np.ndarray]:
    return model.encode(texts)  # (batch, dim) -> 2-D per row

# after
@pw.udf
def embed(texts: list[str]) -> list[list[float]]:
    return [v.reshape(-1).tolist() for v in model.encode(texts)]

Defensive patterns

Strategy: validation

Validate before calling

sample = pw.debug.compute_and_print(table.select(emb=table.emb))
# then, on any materialized value v:
assert np.asarray(v).ndim == 1, f"expected 1-D embedding, got shape {np.asarray(v).shape}"

Type guard

def is_1d_array(v) -> bool:
    return isinstance(v, np.ndarray) and v.ndim == 1

Try / catch

try:
    pw.io.milvus.write(table, uri, collection_name="docs", primary_key=table.id)
except ValueError as e:
    if "dimensional numpy array" in str(e):
        table = table.with_columns(emb=table.emb.apply(lambda a: np.asarray(a).reshape(-1).tolist()))
    else:
        raise

Prevention

Have embedding UDFs return one flat vector per row (list[float] or reshape(-1))
Materialize a sample row with pw.debug.compute_and_print and check ndim before wiring the sink
Type UDF outputs explicitly so shapes surface at graph-build time

When it happens

Trigger: Writing a table whose embedding column holds np.ndarray values with shape (n, m) — e.g. batched embeddings, a 2-D matrix per row, or an accidentally nested array produced by an UDF return type.

Common situations: Embedding UDFs that return the whole batch (shape (batch, dim)) per row; stacking vectors with np.stack and forgetting to index; model outputs reshaped to 2-D.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/0567cfcda9cc5dfa. Report an issue: GitHub.