pathwaycom/pathway · error · ValueError

vector column {name!r} is nullable (type {dtype}); every row

Error message

vector column {name!r} is nullable (type {dtype}); every row must carry a vector, so the column cannot be optional.

What it means

Every Pinecone record must carry a vector, so pw.io.pinecone.write rejects a vector column typed Optional. The check mirrors the runtime PineconeError::InvalidVector guard but fires at write() time, catching the statically-known nullable case before the pipeline starts. Statically-unknown dtypes are skipped.

Source

Thrown at python/pathway/io/pinecone/__init__.py:76

        isinstance(dtype, dt.Tuple)
        and len(dtype.args) == 2
        and dtype.args[0] == dt.INT
        and dtype.args[1] == dt.FLOAT
    )


def _check_vector_dtype(name: str, dtype: dt.DType) -> None:
    """Reject a ``vector`` that is neither a dense nor a sparse vector column.

    A dense vector is a numeric list / array, a sparse one a
    ``list[tuple[int, float]]`` of ``(index, weight)`` pairs. Mirrors the runtime
    ``PineconeError::InvalidVector`` / ``InvalidSparseVector`` guards so a wrong
    column type fails at ``write()`` time rather than once data flows.
    """
    if _is_statically_unknown(dtype):
        return
    if isinstance(dtype, dt.Optional):
        raise ValueError(
            f"vector column {name!r} is nullable (type {dtype}); every row must "
            "carry a vector, so the column cannot be optional."
        )
    if isinstance(dtype, dt.List):
        inner = dtype.wrapped
        if _is_numeric(inner) or _is_sparse_pair(inner):
            return
        if isinstance(inner, (dt.List, dt.Array)):
            raise NotImplementedError(
                f"vector column {name!r} has type {dtype}, which is a multivector; "
                "a Pinecone record carries a single dense or sparse vector, so "
                "multivectors are not supported."
            )
    if isinstance(dtype, dt.Array) and _is_numeric(dtype.wrapped):
        return
    if isinstance(dtype, dt.Tuple) and all(_is_numeric(arg) for arg in dtype.args):
        return
    raise ValueError(

View on GitHub (pinned to fa2f74a464)

Solutions

Filter out rows without embeddings before the sink: t = t.filter(pw.this.embedding.is_not_none()).
Or compute a fallback embedding / drop the record explicitly so the column dtype becomes non-optional.
Fix the embedding step so it always produces a vector for rows you intend to upsert.

Example fix

# before
pw.io.pinecone.write(docs, "idx", primary_key=docs.id, vector=docs.embedding)
# docs.embedding is Optional

# after
indexed = docs.filter(pw.this.embedding.is_not_none())
pw.io.pinecone.write(indexed, "idx", primary_key=indexed.id, vector=indexed.embedding)

Defensive patterns

Strategy: validation

Validate before calling

import pathway as pw

if isinstance(vector._column.dtype, pw.dt.Optional):
    table = table.filter(pw.this[vector._name].is_not_none())
    vector = table[vector._name]

Type guard

import pathway as pw

def is_required_vector_column(col_ref) -> bool:
    return not isinstance(col_ref._column.dtype, pw.dt.Optional)

Try / catch

try:
    pw.io.pinecone.write(docs, "idx", primary_key=docs.id, vector=docs.embedding)
except ValueError as e:
    if "cannot be optional" in str(e):
        docs = docs.filter(pw.this.embedding.is_not_none())
        pw.io.pinecone.write(docs, "idx", primary_key=docs.id, vector=docs.embedding)
    else:
        raise

Prevention

Design embedding steps to always emit a vector; route failures to a dead-letter table.
Filter is_not_none() on embedding columns as the standard step before vector sinks.
Avoid Optional annotations on columns destined to be ids or vectors.

When it happens

Trigger: Passing vector=table.embedding where the embedding column is Optional — typical after outer joins, optional schema fields, or embedding steps that can yield None (e.g. skipped/failed embeddings).

Common situations: Embedding pipelines where some rows fail embedding and stay None; vector columns joined in via ix/outer joins; schemas generated with | None on all fields.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/6af8782d93f645bd. Report an issue: GitHub.