{"record":{"id":"d4c71faf2f123558","repo":"pathwaycom/pathway","slug":"column-k-r-contains-a-non-finite-value-nan-or-i","errorCode":null,"errorMessage":"Column {k!r} contains a non-finite value (NaN or infinity) in its vector, which cannot be indexed by Milvus.","messagePattern":"Column (.+?) contains a non-finite value \\(NaN or infinity\\) in its vector, which cannot be indexed by Milvus\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/pathway/io/milvus/__init__.py","lineNumber":61,"sourceCode":"                    f\"pw.io.milvus.write only supports 1-D arrays (for \"\n                    f\"FLOAT_VECTOR / BINARY_VECTOR fields).\"\n                )\n            v = v.tolist()\n        elif not isinstance(v, _SUPPORTED_TYPES):\n            raise TypeError(\n                f\"Column {k!r} contains a value of unsupported type \"\n                f\"{type(v).__name__!r}. pw.io.milvus.write supports the \"\n                f\"following Pathway types: int, float, str, bool, pw.Json, \"\n                f\"list[float], bytes, and numpy.ndarray (1-D only).\"\n            )\n        # A FLOAT_VECTOR (list / tuple / 1-D array of floats) with a non-finite\n        # component is silently stored by Milvus and corrupts the index —\n        # distances against NaN/infinity are meaningless. Reject it up front with\n        # a clear, column-named error, as the other vector sinks do.\n        if isinstance(v, (list, tuple)) and any(\n            isinstance(x, float) and not math.isfinite(x) for x in v\n        ):\n            raise ValueError(\n                f\"Column {k!r} contains a non-finite value (NaN or infinity) in \"\n                f\"its vector, which cannot be indexed by Milvus.\"\n            )\n        result[k] = v\n    return result\n\n\ndef _is_milvus_transient_connect_error(e: Exception) -> bool:\n    \"\"\"Whether ``e`` is a local milvus-lite embedded-server connection race.\n\n    milvus-lite reports its local server as started as soon as the server\n    process is alive, before the server's local socket actually accepts\n    connections. The first client therefore races the socket coming up, and a\n    client that loses the race fails with a ``server unavailable`` /\n    ``connect failed`` error. The socket can take a few seconds to appear when\n    many local databases start at once, so this is retried in\n    :func:`_connect_with_retry`.\n    \"\"\"","sourceCodeStart":43,"sourceCodeEnd":79,"githubUrl":"https://github.com/pathwaycom/pathway/blob/fa2f74a4649b7c5908690cf60137263d8d80de5f/python/pathway/io/milvus/__init__.py#L43-L79","documentation":"Milvus accepts NaN/infinity components in FLOAT_VECTOR fields but the resulting index is meaningless — distances computed against non-finite values are garbage. Pathway's connector therefore scans every list/tuple vector before writing and raises ValueError (with the column name) if any float component is not finite, protecting the index from silent corruption.","triggerScenarios":"An embedding column containing a list/tuple (or 1-D array converted to list) where at least one element is float('nan'), float('inf'), or float('-inf').","commonSituations":"Model failures producing NaN embeddings (division by zero, missing inputs); empty string inputs producing NaN from some embedding APIs; upstream joins yielding inf similarity scores stored as vectors.","solutions":["Filter or repair bad embeddings before the sink: table.filter(lambda **kw: all(math.isfinite(x) for x in kw['emb']))","Fix the embedding UDF: sanitize with np.nan_to_num(v) or raise on non-finite outputs","Log offending rows by adding a boolean validity column computed from the vector, then route invalid rows to a dead-letter table"],"exampleFix":"# before\nt = t.with_columns(emb=embed(t.text))  # may produce NaN\n\n# after\nimport numpy as np\nt = t.with_columns(emb=embed(t.text).apply(lambda v: np.nan_to_num(v, nan=0.0).tolist()))","handlingStrategy":"validation","validationCode":"def vector_finite(v) -> bool:\n    return all(math.isfinite(x) for x in v if isinstance(x, float))\n\n# use as a Pathway filter over materialized test data before wiring the sink","typeGuard":"import math\n\ndef is_finite_vector(v) -> bool:\n    try:\n        return all(math.isfinite(float(x)) for x in v)\n    except (TypeError, ValueError):\n        return False","tryCatchPattern":"try:\n    pw.io.milvus.write(table, uri, \"docs\", primary_key=table.id)\nexcept ValueError as e:\n    if \"non-finite value\" in str(e):\n        table = table.with_columns(emb=table.emb.apply(lambda v: np.nan_to_num(np.asarray(v, dtype=float), nan=0.0).tolist()))\n    else:\n        raise","preventionTips":["Sanitize embeddings in the UDF with np.nan_to_num or reject non-finite outputs","Add a validity column (all(math.isfinite(x) for x in v)) and filter bad rows into a dead-letter table","Investigate why embeddings contain NaN — usually a model or input problem worth fixing at source"],"tags":["milvus","pathway","nan","embeddings","data-quality"],"backgroundTag":null,"analyzedSha":"fa2f74a4649b7c5908690cf60137263d8d80de5f","analyzedAt":"2026-08-15T01:48:17.006Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}