pathwaycom/pathway · error · ValueError

primary_key contains duplicate column(s) {sorted(duplicates)

Error message

primary_key contains duplicate column(s) {sorted(duplicates)}. Each column may appear at most once.

What it means

In snapshot mode, pw.io.mssql.write builds a PRIMARY KEY ([x], [x]) clause from the primary_key list; duplicate entries would make SQL Server reject the CREATE TABLE, and worse, the shared SqlQueryTemplate reorders DELETE bindings using the duplicated index so retractions silently bind the wrong values. This ValueError is raised at write() time listing the duplicated column names.

Source

Thrown at python/pathway/io/mssql/__init__.py:457

        table_writer_init_mode=init_mode_from_str(init_mode),
        snapshot_maintenance_on_output=is_snapshot_mode,
    )

    key_field_names = None
    if primary_key is not None:
        # Duplicate entries in `primary_key` produce a nonsensical
        # `PRIMARY KEY ([x], [x])` SQL clause that SQL Server rejects, and
        # the shared `SqlQueryTemplate` reorders DELETE bindings using the
        # duplicated index so retractions silently bind wrong values.
        # Reject with a clear message.
        names_seen: set[str] = set()
        duplicates: list[str] = []
        for pkey in primary_key:
            if pkey.name in names_seen and pkey.name not in duplicates:
                duplicates.append(pkey.name)
            names_seen.add(pkey.name)
        if duplicates:
            raise ValueError(
                f"primary_key contains duplicate column(s) {sorted(duplicates)}. "
                "Each column may appear at most once."
            )
        key_field_names = []
        for pkey in primary_key:
            # Raises ValueError when `pkey` belongs to a different table or
            # does not name a column of `table`, so users get a clear
            # message at write() time instead of an opaque runtime error.
            get_column_index(table, pkey)
            # Reject nullable primary-key columns.  SQL Server refuses to
            # build a PRIMARY KEY on a nullable column, and even if the
            # destination table is hand-crafted to allow NULLs, the MERGE
            # statement uses `target.k = source.k` which is UNKNOWN (not
            # TRUE) when both sides are NULL — so retractions never match.
            if isinstance(pkey._column.dtype, dt.Optional):
                raise ValueError(
                    f"primary_key column {pkey.name!r} is declared nullable "
                    f"({pkey._column.dtype}); primary-key columns must be "

View on GitHub (pinned to fa2f74a464)

Solutions

Deduplicate while preserving order: primary_key=list(dict.fromkeys(primary_key)).
Fix the list construction upstream so the same column is not added twice.
Review composite-key assembly code (unions, config merges) for accidental duplicates.

Example fix

# before
primary_key = tenant_keys + tenant_keys  # accidental duplication
pw.io.mssql.write(t, "t", output_table_type="snapshot", primary_key=primary_key)

# after
primary_key = list(dict.fromkeys(tenant_keys + tenant_keys))
pw.io.mssql.write(t, "t", output_table_type="snapshot", primary_key=primary_key)

Defensive patterns

Strategy: validation

Validate before calling

primary_key = list(dict.fromkeys(primary_key))  # dedupe, preserve order
names = [c._name for c in primary_key]
assert len(names) == len(set(names)), "duplicate primary_key entries"

Type guard

def has_unique_pk_names(primary_key) -> bool:
    names = [c._name for c in primary_key]
    return len(names) == len(set(names))

Try / catch

try:
    pw.io.mssql.write(t, "t", output_table_type="snapshot", primary_key=keys)
except ValueError as e:
    if "duplicate column" in str(e):
        keys = list(dict.fromkeys(keys))
        pw.io.mssql.write(t, "t", output_table_type="snapshot", primary_key=keys)
    else:
        raise

Prevention

Deduplicate key lists with dict.fromkeys() whenever assembling keys from multiple sources.
Keep composite-key definitions in one place instead of concatenating ad-hoc lists.
Duplicated keys corrupt DELETE bindings silently in some templates — never ignore this error.

When it happens

Trigger: Passing the same column reference twice in primary_key, e.g. primary_key=[table.id, table.id], or building the list programmatically so the same column ends up in it more than once.

Common situations: Concatenating key lists from several sources without deduplication (primary_key=keys_a + keys_b); refactoring a composite key list and forgetting to remove the old entry.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/52d4f2f250ecbd30. Report an issue: GitHub.