pathwaycom/pathway · error · ValueError

primary_key references column(s) {foreign} that are not pres

Error message

primary_key references column(s) {foreign} that are not present in the written table; pass ColumnReferences from the table being written, e.g. primary_key=[table.k].

What it means

Raised by pw.io.postgres.write() in snapshot mode when one or more names in primary_key do not exist among the written table's columns. Without the check the engine would emit malformed SQL such as PRIMARY KEY ("unknown") during CREATE TABLE, or an UPSERT that panics at flush — after init_mode may already have modified the destination.

Source

Thrown at python/pathway/io/postgres/__init__.py:918

    if (
        _external_diff_column is not None
        and _external_diff_column._column.dtype != dtype.INT
    ):
        raise ValueError("_external_diff_column can only have an integer type")

    external_diff_column_index = get_column_index(table, _external_diff_column)
    key_field_names = None
    if primary_key is not None:
        key_field_names = [pkey_field.name for pkey_field in primary_key]
        # `primary_key=[other_table.col]` (or a reference whose name
        # simply isn't in `table`) is accepted today but then either
        # generates a malformed CREATE TABLE (``PRIMARY KEY
        # ("unknown")``) on init or produces an UPSERT that panics at
        # flush. Reject up-front with a clear message.
        table_columns = set(table.schema.column_names())
        foreign = sorted(n for n in key_field_names if n not in table_columns)
        if foreign:
            raise ValueError(
                f"primary_key references column(s) {foreign} that are "
                "not present in the written table; pass ColumnReferences "
                "from the table being written, e.g. primary_key=[table.k]."
            )
        # A duplicate reference like `primary_key=[t.k, t.k]` would slip
        # through to `SqlQueryTemplate` and either yield malformed SQL
        # (``PRIMARY KEY ("k", "k")``) on CREATE TABLE or corrupt the
        # ``primary_key_fields`` index-swap used on DELETE. Reject it
        # here with a clear message instead.
        duplicates = sorted(
            {name for name in key_field_names if key_field_names.count(name) > 1}
        )
        if duplicates:
            raise ValueError(f"primary_key contains duplicate column(s) {duplicates}")
        # A nullable primary-key column in snapshot mode is silently
        # broken: either the `NOT NULL` PRIMARY KEY we emit on
        # create_if_not_exists / replace rejects the NULL row at
        # insert time, or (against a pre-existing table that allows

View on GitHub (pinned to fa2f74a464)

Solutions

Pass references from the table being written: primary_key=[table.k].
After renames, update the key reference to the new column name.
If keying on another table's column, select/join it into this table first so the name exists here.

Example fix

# before
pw.io.postgres.write(t, parts, "tbl", output_table_type="snapshot", primary_key=[other.id])
# after
joined = t.join(other, t.id == other.id).select(t.id, t.value, other.extra)
pw.io.postgres.write(joined, parts, "tbl", output_table_type="snapshot", primary_key=[joined.id])

Defensive patterns

Strategy: validation

Validate before calling

table_cols = set(table.schema.column_names())
foreign = [c.name for c in (primary_key or []) if c.name not in table_cols]
assert not foreign, f"primary_key references foreign columns: {foreign}"

Prevention

Always build primary_key from the same table object passed as the first write() argument.
After renames/joins, re-derive key references from the final table.
Add a helper that resolves keys by name against table.schema.column_names() at pipeline-build time.

When it happens

Trigger: primary_key=[other_table.k] where the reference's name is not a column of the table being written; primary_key=[t.col] after renaming that column upstream; passing a plain string that names nothing in the schema.

Common situations: Copy-pasting a key reference from another table variable; joins/rename operations that changed column names between drafting the call and running it.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/16589c0adfb5efe1. Report an issue: GitHub.