pathwaycom/pathway · error · ValueError

primary_key contains duplicate column(s) {duplicates}

Error message

primary_key contains duplicate column(s) {duplicates}

What it means

Raised by pw.io.postgres.write when the primary_key list contains the same column name more than once (e.g. primary_key=[t.k, t.k]). Pathway rejects this eagerly because a duplicate would otherwise flow into the SQL template and produce malformed DDL (PRIMARY KEY (k, k) with the column listed twice) on CREATE TABLE or corrupt the primary_key_fields index swap used for DELETE retractions.

Source

Thrown at python/pathway/io/postgres/__init__.py:932

        # flush. Reject up-front with a clear message.
        table_columns = set(table.schema.column_names())
        foreign = sorted(n for n in key_field_names if n not in table_columns)
        if foreign:
            raise ValueError(
                f"primary_key references column(s) {foreign} that are "
                "not present in the written table; pass ColumnReferences "
                "from the table being written, e.g. primary_key=[table.k]."
            )
        # A duplicate reference like `primary_key=[t.k, t.k]` would slip
        # through to `SqlQueryTemplate` and either yield malformed SQL
        # (``PRIMARY KEY ("k", "k")``) on CREATE TABLE or corrupt the
        # ``primary_key_fields`` index-swap used on DELETE. Reject it
        # here with a clear message instead.
        duplicates = sorted(
            {name for name in key_field_names if key_field_names.count(name) > 1}
        )
        if duplicates:
            raise ValueError(f"primary_key contains duplicate column(s) {duplicates}")
        # A nullable primary-key column in snapshot mode is silently
        # broken: either the `NOT NULL` PRIMARY KEY we emit on
        # create_if_not_exists / replace rejects the NULL row at
        # insert time, or (against a pre-existing table that allows
        # NULL in the PK) the retraction ``DELETE ... WHERE pkey=$1``
        # never matches anything because SQL ``= NULL`` is always
        # false. Both are data-loss footguns, so we refuse the setup
        # here with an actionable message.
        if is_snapshot_mode:
            for pkey_field in primary_key:
                if isinstance(pkey_field._column.dtype, dtype.Optional):
                    raise ValueError(
                        f"primary_key column '{pkey_field.name}' is "
                        "declared nullable; primary_key columns must be "
                        "non-nullable in snapshot mode. Either remove "
                        "the Optional wrapper in the schema or filter "
                        "out nulls upstream via "
                        ".filter(t.pkey.is_not_none())."

View on GitHub (pinned to fa2f74a464)

Solutions

1. Inspect the primary_key argument and remove the duplicate ColumnReference so each column appears exactly once.
2. If the key list is built dynamically, deduplicate while preserving order, e.g. list(dict.fromkeys(key_cols)) before passing it to write().
3. If you intended a composite key, verify each element is a distinct column from the table being written (table.k, table.name, ...).

Example fix

# before
pw.io.postgres.write(t, conn, "t", primary_key=[t.k, t.k])

# after
pw.io.postgres.write(t, conn, "t", primary_key=[t.k])

key_cols = [t.k, t.name]
names = [c.name() for c in key_cols]
assert len(set(names)) == len(names), f"duplicate primary_key columns: {names}"
pw.io.postgres.write(t, conn, "t", primary_key=key_cols)

def unique_key_columns(cols: list) -> bool:
    names = [c.name() for c in cols]
    return len(set(names)) == len(names)

try:
    pw.io.postgres.write(t, conn, "t", primary_key=keys)
except ValueError as e:
    if "duplicate column" in str(e):
        keys = list(dict.fromkeys(keys))
        pw.io.postgres.write(t, conn, "t", primary_key=keys)
    else:
        raise

When it happens

Trigger: Calling pw.io.postgres.write(table, ..., primary_key=[t.k, t.k]) or building the primary_key list dynamically in a way that appends the same ColumnReference twice (e.g. primary_key=keys + keys, or a loop that adds a column per sort key plus the explicit key again).

Common situations: Composite primary keys assembled programmatically from several sources where deduplication was forgotten; copy-paste of a key column into a key list; passing sorted(columns) where a column appears in two roles.

Related errors

• primary key field names must be specified for a snapshot mod
• Incorrect S3 path: {s3_path}
• SchemaRegistryHeader.key must be a str, got {type(self.key).
• primary_key column {primary_key._name!r} does not belong to
• Defining a primary key in the schema is not supported for pw

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/c9a11f754a7c3d16. Report an issue: GitHub.