{"record":{"id":"9a6e62e43088a720","repo":"pathwaycom/pathway","slug":"primary-key-column-name-r-is-nullable-type-dty","errorCode":null,"errorMessage":"primary_key column {name!r} is nullable (type {dtype}); a Pinecone record id must always be present, so the column cannot be optional.","messagePattern":"primary_key column (.+?) is nullable \\(type (.+?)\\); a Pinecone record id must always be present, so the column cannot be optional\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/pathway/io/pinecone/__init__.py","lineNumber":43,"sourceCode":"    return dtype == dt.ANY\n\n\ndef _is_numeric(dtype: dt.DType) -> bool:\n    return dtype in (dt.INT, dt.FLOAT) or _is_statically_unknown(dtype)\n\n\ndef _check_primary_key_dtype(name: str, dtype: dt.DType) -> None:\n    \"\"\"Reject a ``primary_key`` whose type can never be a Pinecone record id.\n\n    The id must always be present and must be an ``int`` or ``str`` (a pointer\n    is accepted too, since the engine stringifies it). The matching runtime\n    guard (``PineconeError::InvalidId``) only fires once an offending row reaches\n    the sink, so catch the statically-known cases at ``write()`` time.\n    \"\"\"\n    if _is_statically_unknown(dtype):\n        return\n    if isinstance(dtype, dt.Optional):\n        raise ValueError(\n            f\"primary_key column {name!r} is nullable (type {dtype}); a Pinecone \"\n            \"record id must always be present, so the column cannot be optional.\"\n        )\n    if dtype in (dt.INT, dt.STR) or isinstance(dtype, dt.Pointer):\n        return\n    raise ValueError(\n        f\"primary_key column {name!r} has unsupported type {dtype}; a Pinecone \"\n        \"record id must be int or str.\"\n    )\n\n\ndef _is_sparse_pair(dtype: dt.DType) -> bool:\n    \"\"\"Whether ``dtype`` is the ``tuple[int, float]`` of a sparse (index, weight) pair.\"\"\"\n    return (\n        isinstance(dtype, dt.Tuple)\n        and len(dtype.args) == 2\n        and dtype.args[0] == dt.INT\n        and dtype.args[1] == dt.FLOAT","sourceCodeStart":25,"sourceCodeEnd":61,"githubUrl":"https://github.com/pathwaycom/pathway/blob/fa2f74a4649b7c5908690cf60137263d8d80de5f/python/pathway/io/pinecone/__init__.py#L25-L61","documentation":"A Pinecone record id must always be present, so pw.io.pinecone.write rejects a primary_key column whose dtype is Optional. The runtime guard (PineconeError::InvalidId) would only fire once an offending row reaches the sink; this call-time ValueError catches the statically-known nullable case early. Columns whose type is not statically known are skipped.","triggerScenarios":"Passing primary_key=table.doc_id to pw.io.pinecone.write where table.doc_id has an Optional dtype — e.g. it came from an outer join, an optional schema column, or a computed column that may be None.","commonSituations":"Using a join output column as the Pinecone id; schemas with | None annotations on the document id; ids produced by optional lookups (pw.Table.filter + outer join) without an unwrap step.","solutions":["Make the key column non-nullable before write(), e.g. filter out rows with missing ids or pw.coalesce() to a fallback id.","Use a column that is never None, such as this_row.id or the ingested document's required id column.","If rows legitimately lack ids, decide explicitly whether to drop them (filter) or synthesize ids before sinking."],"exampleFix":"# before\npw.io.pinecone.write(docs, \"my-index\", primary_key=docs.doc_id, vector=docs.embedding)\n# docs.doc_id is Optional\n\n# after\nupsertable = docs.filter(pw.this.doc_id.is_not_none())\npw.io.pinecone.write(upsertable, \"my-index\", primary_key=upsertable.doc_id, vector=upsertable.embedding)","handlingStrategy":"validation","validationCode":"import pathway as pw\n\nif isinstance(primary_key._column.dtype, pw.dt.Optional):\n    table = table.filter(pw.this[primary_key._name].is_not_none())\n    primary_key = table[primary_key._name]","typeGuard":"import pathway as pw\n\ndef is_required_column(col_ref) -> bool:\n    return not isinstance(col_ref._column.dtype, pw.dt.Optional)","tryCatchPattern":"try:\n    pw.io.pinecone.write(docs, \"idx\", primary_key=docs.doc_id, vector=docs.vec)\nexcept ValueError as e:\n    if \"cannot be optional\" in str(e):\n        docs = docs.filter(pw.this.doc_id.is_not_none())\n        pw.io.pinecone.write(docs, \"idx\", primary_key=docs.doc_id, vector=docs.vec)\n    else:\n        raise","preventionTips":["Filter rows with missing ids before vector sinks instead of relying on runtime guards.","Prefer this_row.id or a source-required column as the Pinecone primary key.","Check join-derived key columns for Optionality before passing them as primary_key."],"tags":["pinecone","primary-key","nullable","vector-db","pathway"],"backgroundTag":null,"analyzedSha":"fa2f74a4649b7c5908690cf60137263d8d80de5f","analyzedAt":"2026-08-15T01:48:17.006Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}