pathwaycom/pathway · error · ValueError

Column {k!r} contains a non-finite value (NaN or infinity) i

Error message

Column {k!r} contains a non-finite value (NaN or infinity) in its vector, which cannot be indexed by Milvus.

What it means

Milvus accepts NaN/infinity components in FLOAT_VECTOR fields but the resulting index is meaningless — distances computed against non-finite values are garbage. Pathway's connector therefore scans every list/tuple vector before writing and raises ValueError (with the column name) if any float component is not finite, protecting the index from silent corruption.

Source

Thrown at python/pathway/io/milvus/__init__.py:61

                    f"pw.io.milvus.write only supports 1-D arrays (for "
                    f"FLOAT_VECTOR / BINARY_VECTOR fields)."
                )
            v = v.tolist()
        elif not isinstance(v, _SUPPORTED_TYPES):
            raise TypeError(
                f"Column {k!r} contains a value of unsupported type "
                f"{type(v).__name__!r}. pw.io.milvus.write supports the "
                f"following Pathway types: int, float, str, bool, pw.Json, "
                f"list[float], bytes, and numpy.ndarray (1-D only)."
            )
        # A FLOAT_VECTOR (list / tuple / 1-D array of floats) with a non-finite
        # component is silently stored by Milvus and corrupts the index —
        # distances against NaN/infinity are meaningless. Reject it up front with
        # a clear, column-named error, as the other vector sinks do.
        if isinstance(v, (list, tuple)) and any(
            isinstance(x, float) and not math.isfinite(x) for x in v
        ):
            raise ValueError(
                f"Column {k!r} contains a non-finite value (NaN or infinity) in "
                f"its vector, which cannot be indexed by Milvus."
            )
        result[k] = v
    return result


def _is_milvus_transient_connect_error(e: Exception) -> bool:
    """Whether ``e`` is a local milvus-lite embedded-server connection race.

    milvus-lite reports its local server as started as soon as the server
    process is alive, before the server's local socket actually accepts
    connections. The first client therefore races the socket coming up, and a
    client that loses the race fails with a ``server unavailable`` /
    ``connect failed`` error. The socket can take a few seconds to appear when
    many local databases start at once, so this is retried in
    :func:`_connect_with_retry`.
    """

View on GitHub (pinned to fa2f74a464)

Solutions

Filter or repair bad embeddings before the sink: table.filter(lambda **kw: all(math.isfinite(x) for x in kw['emb']))
Fix the embedding UDF: sanitize with np.nan_to_num(v) or raise on non-finite outputs
Log offending rows by adding a boolean validity column computed from the vector, then route invalid rows to a dead-letter table

Example fix

# before
t = t.with_columns(emb=embed(t.text))  # may produce NaN

# after
import numpy as np
t = t.with_columns(emb=embed(t.text).apply(lambda v: np.nan_to_num(v, nan=0.0).tolist()))

Defensive patterns

Strategy: validation

Validate before calling

def vector_finite(v) -> bool:
    return all(math.isfinite(x) for x in v if isinstance(x, float))

# use as a Pathway filter over materialized test data before wiring the sink

Type guard

import math

def is_finite_vector(v) -> bool:
    try:
        return all(math.isfinite(float(x)) for x in v)
    except (TypeError, ValueError):
        return False

Try / catch

try:
    pw.io.milvus.write(table, uri, "docs", primary_key=table.id)
except ValueError as e:
    if "non-finite value" in str(e):
        table = table.with_columns(emb=table.emb.apply(lambda v: np.nan_to_num(np.asarray(v, dtype=float), nan=0.0).tolist()))
    else:
        raise

Prevention

Sanitize embeddings in the UDF with np.nan_to_num or reject non-finite outputs
Add a validity column (all(math.isfinite(x) for x in v)) and filter bad rows into a dead-letter table
Investigate why embeddings contain NaN — usually a model or input problem worth fixing at source

When it happens

Trigger: An embedding column containing a list/tuple (or 1-D array converted to list) where at least one element is float('nan'), float('inf'), or float('-inf').

Common situations: Model failures producing NaN embeddings (division by zero, missing inputs); empty string inputs producing NaN from some embedding APIs; upstream joins yielding inf similarity scores stored as vectors.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/d4c71faf2f123558. Report an issue: GitHub.