pathwaycom/pathway · error · ValueError

primary_key column {name!r} is nullable (type {dtype}); a Pi

Error message

primary_key column {name!r} is nullable (type {dtype}); a Pinecone record id must always be present, so the column cannot be optional.

What it means

A Pinecone record id must always be present, so pw.io.pinecone.write rejects a primary_key column whose dtype is Optional. The runtime guard (PineconeError::InvalidId) would only fire once an offending row reaches the sink; this call-time ValueError catches the statically-known nullable case early. Columns whose type is not statically known are skipped.

Source

Thrown at python/pathway/io/pinecone/__init__.py:43

    return dtype == dt.ANY


def _is_numeric(dtype: dt.DType) -> bool:
    return dtype in (dt.INT, dt.FLOAT) or _is_statically_unknown(dtype)


def _check_primary_key_dtype(name: str, dtype: dt.DType) -> None:
    """Reject a ``primary_key`` whose type can never be a Pinecone record id.

    The id must always be present and must be an ``int`` or ``str`` (a pointer
    is accepted too, since the engine stringifies it). The matching runtime
    guard (``PineconeError::InvalidId``) only fires once an offending row reaches
    the sink, so catch the statically-known cases at ``write()`` time.
    """
    if _is_statically_unknown(dtype):
        return
    if isinstance(dtype, dt.Optional):
        raise ValueError(
            f"primary_key column {name!r} is nullable (type {dtype}); a Pinecone "
            "record id must always be present, so the column cannot be optional."
        )
    if dtype in (dt.INT, dt.STR) or isinstance(dtype, dt.Pointer):
        return
    raise ValueError(
        f"primary_key column {name!r} has unsupported type {dtype}; a Pinecone "
        "record id must be int or str."
    )


def _is_sparse_pair(dtype: dt.DType) -> bool:
    """Whether ``dtype`` is the ``tuple[int, float]`` of a sparse (index, weight) pair."""
    return (
        isinstance(dtype, dt.Tuple)
        and len(dtype.args) == 2
        and dtype.args[0] == dt.INT
        and dtype.args[1] == dt.FLOAT

View on GitHub (pinned to fa2f74a464)

Solutions

1. Make the key column non-nullable before write(), e.g. filter out rows with missing ids or pw.coalesce() to a fallback id.
2. Use a column that is never None, such as this_row.id or the ingested document's required id column.
3. If rows legitimately lack ids, decide explicitly whether to drop them (filter) or synthesize ids before sinking.

Example fix

# before
pw.io.pinecone.write(docs, "my-index", primary_key=docs.doc_id, vector=docs.embedding)
# docs.doc_id is Optional

# after
upsertable = docs.filter(pw.this.doc_id.is_not_none())
pw.io.pinecone.write(upsertable, "my-index", primary_key=upsertable.doc_id, vector=upsertable.embedding)

import pathway as pw

if isinstance(primary_key._column.dtype, pw.dt.Optional):
    table = table.filter(pw.this[primary_key._name].is_not_none())
    primary_key = table[primary_key._name]

import pathway as pw

def is_required_column(col_ref) -> bool:
    return not isinstance(col_ref._column.dtype, pw.dt.Optional)

try:
    pw.io.pinecone.write(docs, "idx", primary_key=docs.doc_id, vector=docs.vec)
except ValueError as e:
    if "cannot be optional" in str(e):
        docs = docs.filter(pw.this.doc_id.is_not_none())
        pw.io.pinecone.write(docs, "idx", primary_key=docs.doc_id, vector=docs.vec)
    else:
        raise

When it happens

Trigger: Passing primary_key=table.doc_id to pw.io.pinecone.write where table.doc_id has an Optional dtype — e.g. it came from an outer join, an optional schema column, or a computed column that may be None.

Common situations: Using a join output column as the Pinecone id; schemas with | None annotations on the document id; ids produced by optional lookups (pw.Table.filter + outer join) without an unwrap step.

Related errors

• primary_key column {name!r} has unsupported type {dtype}; a
• pw.io.mssql.read primary_key column(s) {nullable_pks} are de
• primary_key column {pkey.name!r} is declared nullable ({pkey
• pw.io.mysql.read primary_key column(s) {nullable_pks} are de
• vector column {name!r} is nullable (type {dtype}); every row

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/9a6e62e43088a720. Report an issue: GitHub.