pathwaycom/pathway · error · ValueError

primary_key contains duplicate column(s) {sorted(duplicates)

Error message

primary_key contains duplicate column(s) {sorted(duplicates)}. Each column may appear at most once.

What it means

Raised by pw.io.duckdb.write when the primary_key sequence contains the same column more than once. Duplicate key entries produce a nonsensical SQL template and reorder DELETE bindings using the duplicated index, so retractions would silently match no rows; the connector rejects duplicates up front.

Source

Thrown at python/pathway/io/duckdb/__init__.py:336

                f"Column(s) {collisions} collide with the 'time' and 'diff' "
                "metadata columns appended in stream_of_changes mode. Rename "
                "these columns in the Pathway table, or use "
                'output_table_type="snapshot".'
            )

    key_field_names: list[str] | None = None
    if primary_key is not None:
        # Duplicate entries in `primary_key` produce a nonsensical SQL template
        # and reorder DELETE bindings using the duplicated index, so retractions
        # would silently match no rows. Reject here with a clear message.
        names_seen: set[str] = set()
        duplicates: list[str] = []
        for pkey in primary_key:
            if pkey.name in names_seen and pkey.name not in duplicates:
                duplicates.append(pkey.name)
            names_seen.add(pkey.name)
        if duplicates:
            raise ValueError(
                f"primary_key contains duplicate column(s) {sorted(duplicates)}. "
                "Each column may appear at most once."
            )
        key_field_names = []
        for pkey in primary_key:
            # Raises ValueError when `pkey` belongs to a different table or does
            # not name a column of `table`, so users get a clear message at
            # write() time instead of an opaque runtime error.
            get_column_index(table, pkey)
            # A nullable primary key makes DELETE ... WHERE pk = NULL never match
            # on retractions, so the destination would keep stale rows.
            if isinstance(pkey._column.dtype, dt.Optional):
                raise ValueError(
                    f"primary_key column {pkey.name!r} is declared nullable "
                    f"({pkey._column.dtype}); primary-key columns must be "
                    "non-nullable in snapshot mode."
                )
            # DuckDB cannot build a PRIMARY KEY / index on a list, array, tuple or

View on GitHub (pinned to fa2f74a464)

Solutions

1. Deduplicate the key list while preserving order: primary_key=list(dict.fromkeys([t.id, t.tenant_id, t.id])).
2. Remove the repeated column from the primary_key list manually.

Example fix

# before
keys = [*config_keys, *default_keys]  # t.id appears twice
pw.io.duckdb.write(t, table_name="t", output_table_type="snapshot", primary_key=keys)

# after
keys = list(dict.fromkeys([*config_keys, *default_keys]))
pw.io.duckdb.write(t, table_name="t", output_table_type="snapshot", primary_key=keys)

primary_key = list(dict.fromkeys(primary_key))  # order-preserving dedup

When it happens

Trigger: pw.io.duckdb.write(t, table_name="t", output_table_type="snapshot", primary_key=[t.id, t.id]) — typically when a key list is built dynamically (e.g. [*default_keys, *extra_keys]) and the same column ends up in both.

Common situations: Programmatically composing composite keys from config plus hardcoded columns; concatenating key lists without deduplication after a refactor.

Related errors

• primary_key can only be specified for the snapshot table typ
• primary_key must be specified for the snapshot table type
• database {path_str!r} is an existing directory, not a DuckDB
• sort_by cannot be used with the snapshot table type: a snaps
• detach_between_batches=True cannot be used with database=":m

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/b4e15080bcab3cd9. Report an issue: GitHub.