{"record":{"id":"6af8782d93f645bd","repo":"pathwaycom/pathway","slug":"vector-column-name-r-is-nullable-type-dtype","errorCode":null,"errorMessage":"vector column {name!r} is nullable (type {dtype}); every row must carry a vector, so the column cannot be optional.","messagePattern":"vector column (.+?) is nullable \\(type (.+?)\\); every row must carry a vector, so the column cannot be optional\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/pathway/io/pinecone/__init__.py","lineNumber":76,"sourceCode":"        isinstance(dtype, dt.Tuple)\n        and len(dtype.args) == 2\n        and dtype.args[0] == dt.INT\n        and dtype.args[1] == dt.FLOAT\n    )\n\n\ndef _check_vector_dtype(name: str, dtype: dt.DType) -> None:\n    \"\"\"Reject a ``vector`` that is neither a dense nor a sparse vector column.\n\n    A dense vector is a numeric list / array, a sparse one a\n    ``list[tuple[int, float]]`` of ``(index, weight)`` pairs. Mirrors the runtime\n    ``PineconeError::InvalidVector`` / ``InvalidSparseVector`` guards so a wrong\n    column type fails at ``write()`` time rather than once data flows.\n    \"\"\"\n    if _is_statically_unknown(dtype):\n        return\n    if isinstance(dtype, dt.Optional):\n        raise ValueError(\n            f\"vector column {name!r} is nullable (type {dtype}); every row must \"\n            \"carry a vector, so the column cannot be optional.\"\n        )\n    if isinstance(dtype, dt.List):\n        inner = dtype.wrapped\n        if _is_numeric(inner) or _is_sparse_pair(inner):\n            return\n        if isinstance(inner, (dt.List, dt.Array)):\n            raise NotImplementedError(\n                f\"vector column {name!r} has type {dtype}, which is a multivector; \"\n                \"a Pinecone record carries a single dense or sparse vector, so \"\n                \"multivectors are not supported.\"\n            )\n    if isinstance(dtype, dt.Array) and _is_numeric(dtype.wrapped):\n        return\n    if isinstance(dtype, dt.Tuple) and all(_is_numeric(arg) for arg in dtype.args):\n        return\n    raise ValueError(","sourceCodeStart":58,"sourceCodeEnd":94,"githubUrl":"https://github.com/pathwaycom/pathway/blob/fa2f74a4649b7c5908690cf60137263d8d80de5f/python/pathway/io/pinecone/__init__.py#L58-L94","documentation":"Every Pinecone record must carry a vector, so pw.io.pinecone.write rejects a vector column typed Optional. The check mirrors the runtime PineconeError::InvalidVector guard but fires at write() time, catching the statically-known nullable case before the pipeline starts. Statically-unknown dtypes are skipped.","triggerScenarios":"Passing vector=table.embedding where the embedding column is Optional — typical after outer joins, optional schema fields, or embedding steps that can yield None (e.g. skipped/failed embeddings).","commonSituations":"Embedding pipelines where some rows fail embedding and stay None; vector columns joined in via ix/outer joins; schemas generated with | None on all fields.","solutions":["Filter out rows without embeddings before the sink: t = t.filter(pw.this.embedding.is_not_none()).","Or compute a fallback embedding / drop the record explicitly so the column dtype becomes non-optional.","Fix the embedding step so it always produces a vector for rows you intend to upsert."],"exampleFix":"# before\npw.io.pinecone.write(docs, \"idx\", primary_key=docs.id, vector=docs.embedding)\n# docs.embedding is Optional\n\n# after\nindexed = docs.filter(pw.this.embedding.is_not_none())\npw.io.pinecone.write(indexed, \"idx\", primary_key=indexed.id, vector=indexed.embedding)","handlingStrategy":"validation","validationCode":"import pathway as pw\n\nif isinstance(vector._column.dtype, pw.dt.Optional):\n    table = table.filter(pw.this[vector._name].is_not_none())\n    vector = table[vector._name]","typeGuard":"import pathway as pw\n\ndef is_required_vector_column(col_ref) -> bool:\n    return not isinstance(col_ref._column.dtype, pw.dt.Optional)","tryCatchPattern":"try:\n    pw.io.pinecone.write(docs, \"idx\", primary_key=docs.id, vector=docs.embedding)\nexcept ValueError as e:\n    if \"cannot be optional\" in str(e):\n        docs = docs.filter(pw.this.embedding.is_not_none())\n        pw.io.pinecone.write(docs, \"idx\", primary_key=docs.id, vector=docs.embedding)\n    else:\n        raise","preventionTips":["Design embedding steps to always emit a vector; route failures to a dead-letter table.","Filter is_not_none() on embedding columns as the standard step before vector sinks.","Avoid Optional annotations on columns destined to be ids or vectors."],"tags":["pinecone","vector","nullable","embeddings","pathway"],"backgroundTag":null,"analyzedSha":"fa2f74a4649b7c5908690cf60137263d8d80de5f","analyzedAt":"2026-08-15T01:48:17.006Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}