{"record":{"id":"0567cfcda9cc5dfa","repo":"pathwaycom/pathway","slug":"column-k-r-contains-a-v-ndim-dimensional-numpy","errorCode":null,"errorMessage":"Column {k!r} contains a {v.ndim}-dimensional numpy array. pw.io.milvus.write only supports 1-D arrays (for FLOAT_VECTOR / BINARY_VECTOR fields).","messagePattern":"Column (.+?) contains a (.+?)-dimensional numpy array\\. pw\\.io\\.milvus\\.write only supports 1-D arrays \\(for FLOAT_VECTOR / BINARY_VECTOR fields\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/pathway/io/milvus/__init__.py","lineNumber":41,"sourceCode":"_SUPPORTED_TYPES = (bool, int, float, str, dict, list, tuple, bytes, np.ndarray)\n\n\ndef _prepare_row(row: dict) -> dict:\n    \"\"\"Convert Pathway Live Data Framework-internal types to plain Python values for pymilvus.\n\n    Unwraps ``pw.Json`` wrapper objects, converts 1-D ``numpy.ndarray`` values\n    to lists, and validates that every value belongs to a type the Milvus\n    connector supports.  Raises ``TypeError`` with a descriptive message for\n    unsupported types, and ``ValueError`` for multi-dimensional arrays or for a\n    vector containing a non-finite (NaN / infinity) component.\n    \"\"\"\n    result = {}\n    for k, v in row.items():\n        if isinstance(v, _PwJson):\n            v = v.value\n        if isinstance(v, np.ndarray):\n            if v.ndim != 1:\n                raise ValueError(\n                    f\"Column {k!r} contains a {v.ndim}-dimensional numpy array. \"\n                    f\"pw.io.milvus.write only supports 1-D arrays (for \"\n                    f\"FLOAT_VECTOR / BINARY_VECTOR fields).\"\n                )\n            v = v.tolist()\n        elif not isinstance(v, _SUPPORTED_TYPES):\n            raise TypeError(\n                f\"Column {k!r} contains a value of unsupported type \"\n                f\"{type(v).__name__!r}. pw.io.milvus.write supports the \"\n                f\"following Pathway types: int, float, str, bool, pw.Json, \"\n                f\"list[float], bytes, and numpy.ndarray (1-D only).\"\n            )\n        # A FLOAT_VECTOR (list / tuple / 1-D array of floats) with a non-finite\n        # component is silently stored by Milvus and corrupts the index —\n        # distances against NaN/infinity are meaningless. Reject it up front with\n        # a clear, column-named error, as the other vector sinks do.\n        if isinstance(v, (list, tuple)) and any(\n            isinstance(x, float) and not math.isfinite(x) for x in v","sourceCodeStart":23,"sourceCodeEnd":59,"githubUrl":"https://github.com/pathwaycom/pathway/blob/fa2f74a4649b7c5908690cf60137263d8d80de5f/python/pathway/io/milvus/__init__.py#L23-L59","documentation":"Milvus vector fields (FLOAT_VECTOR, BINARY_VECTOR) are flat, one-dimensional vectors. When a row handed to pw.io.milvus.write contains a numpy array with more than one dimension (or zero), Pathway raises ValueError naming the column and the offending dimensionality instead of letting Milvus silently corrupt or reject the data.","triggerScenarios":"Writing a table whose embedding column holds np.ndarray values with shape (n, m) — e.g. batched embeddings, a 2-D matrix per row, or an accidentally nested array produced by an UDF return type.","commonSituations":"Embedding UDFs that return the whole batch (shape (batch, dim)) per row; stacking vectors with np.stack and forgetting to index; model outputs reshaped to 2-D.","solutions":["Fix the UDF to return one 1-D vector per row (index the batch: embeddings[i])","Flatten/validate in the transform step: pw.this.emb.apply(lambda a: np.asarray(a).reshape(-1))","Inspect shapes before writing: assert np.asarray(v).ndim == 1 for sampled values"],"exampleFix":"# before\n@pw.udf\ndef embed(texts: list[str]) -> list[np.ndarray]:\n    return model.encode(texts)  # (batch, dim) -> 2-D per row\n\n# after\n@pw.udf\ndef embed(texts: list[str]) -> list[list[float]]:\n    return [v.reshape(-1).tolist() for v in model.encode(texts)]","handlingStrategy":"validation","validationCode":"sample = pw.debug.compute_and_print(table.select(emb=table.emb))\n# then, on any materialized value v:\nassert np.asarray(v).ndim == 1, f\"expected 1-D embedding, got shape {np.asarray(v).shape}\"","typeGuard":"def is_1d_array(v) -> bool:\n    return isinstance(v, np.ndarray) and v.ndim == 1","tryCatchPattern":"try:\n    pw.io.milvus.write(table, uri, collection_name=\"docs\", primary_key=table.id)\nexcept ValueError as e:\n    if \"dimensional numpy array\" in str(e):\n        table = table.with_columns(emb=table.emb.apply(lambda a: np.asarray(a).reshape(-1).tolist()))\n    else:\n        raise","preventionTips":["Have embedding UDFs return one flat vector per row (list[float] or reshape(-1))","Materialize a sample row with pw.debug.compute_and_print and check ndim before wiring the sink","Type UDF outputs explicitly so shapes surface at graph-build time"],"tags":["milvus","pathway","numpy","embeddings","validation"],"backgroundTag":null,"analyzedSha":"fa2f74a4649b7c5908690cf60137263d8d80de5f","analyzedAt":"2026-08-15T01:48:17.006Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}