pathwaycom/pathway · error · ValueError

primary_key column {primary_key._name!r} does not belong to

Error message

primary_key column {primary_key._name!r} does not belong to the provided table. Pass a column reference from the same table, e.g. primary_key=table.{primary_key._name}.

What it means

pw.io.pinecone.write requires the primary_key column reference to come from the exact table being written; a reference from any other table (or a column of a different pipeline stage) cannot be evaluated in this sink's context. The connector checks primary_key._table is not table at call time and raises this ValueError with a suggested corrected form.

Source

Thrown at python/pathway/io/pinecone/__init__.py:340

    ...     primary_key=table.doc_id,
    ...     vector=table.embedding,
    ...     metadata_columns=[table.title],
    ...     api_key="YOUR_API_KEY",
    ... )
    >>> pw.io.pinecone.write(   # doctest: +SKIP
    ...     table,
    ...     index_name="docs-sparse",
    ...     primary_key=table.doc_id,
    ...     vector=table.bm25,
    ...     metadata_columns=[table.title],
    ...     api_key="YOUR_API_KEY",
    ... )
    >>> pw.run(monitoring_level=pw.MonitoringLevel.NONE)  # doctest: +SKIP
    """
    _check_entitlements("pinecone")

    if primary_key is not None and primary_key._table is not table:
        raise ValueError(
            f"primary_key column {primary_key._name!r} does not belong to the "
            f"provided table. Pass a column reference from the same table, "
            f"e.g. primary_key=table.{primary_key._name}."
        )
    if vector._table is not table:
        raise ValueError(
            f"vector column {vector._name!r} does not belong to the provided "
            f"table. Pass a column reference from the same table, "
            f"e.g. vector=table.{vector._name}."
        )

    pk_name = primary_key._name if primary_key is not None else None
    vector_name = vector._name

    if pk_name == vector_name:
        raise ValueError(
            f"primary_key and vector both reference column {pk_name!r}; they must "
            "be different columns."

View on GitHub (pinned to fa2f74a464)

Solutions

Re-take the column reference from the exact table you pass as the first argument: primary_key=table.doc_id (same object).
Store the final table in one variable and reference all sink columns from it after all transformations.
Note that filter/rename/with_columns return new tables — always re-index column references from the latest table.

Example fix

# before
base = pw.io.jsonlines.read("in.jsonl", schema=InSchema)
key_ref = base.doc_id
enriched = base.with_columns(score=pw.this.text.len())
pw.io.pinecone.write(enriched, "idx", primary_key=key_ref, vector=enriched.vec)

# after
enriched = base.with_columns(score=pw.this.text.len())
pw.io.pinecone.write(enriched, "idx", primary_key=enriched.doc_id, vector=enriched.vec)

Defensive patterns

Strategy: type-guard

Validate before calling

def belongs_to(col_ref, table) -> bool:
    return col_ref._table is table

assert primary_key is None or belongs_to(primary_key, table), (
    f"primary_key must reference a column of the same table, got {primary_key._name!r}"
)
assert belongs_to(vector, table), "vector must reference a column of the same table"

Type guard

def is_column_of(col_ref, table) -> bool:
    """True when col_ref belongs to exactly this Table instance."""
    return getattr(col_ref, "_table", None) is table

Try / catch

try:
    pw.io.pinecone.write(enriched, "idx", primary_key=key_ref, vector=enriched.vec)
except ValueError as e:
    if "does not belong to" in str(e):
        pw.io.pinecone.write(enriched, "idx", primary_key=enriched.doc_id, vector=enriched.vec)
    else:
        raise

Prevention

Always re-derive column references from the final table object after transformations.
Keep one variable per pipeline stage; never mix references across stages in sink calls.
Remember Pathway transformations (filter/with_columns/select) return new Table objects.

When it happens

Trigger: Passing primary_key=some_other_table.id or a column reference captured before a filter/join/with_columns chain produced a new table object, while the first positional argument is a different table object.

Common situations: Reusing a column reference variable from an earlier pipeline stage after the table was transformed (transformations return new Table objects); copy-pasting write() calls and mixing references between two tables; passing a source-table key alongside a joined/enriched table.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/1e7f5449baf20340. Report an issue: GitHub.