pathwaycom/pathway · error · ValueError

primary_key contains duplicate column(s) {sorted(duplicates)

Error message

primary_key contains duplicate column(s) {sorted(duplicates)}. Each column may appear at most once.

What it means

Duplicate entries in primary_key (e.g. [t.k, t.k]) would generate a nonsensical PRIMARY KEY ("k", "k") clause which SQLite itself tolerates, but Pathway's shared SqlQueryTemplate then reorders DELETE bindings using the duplicated index and retractions silently match no rows — a silent data-corruption bug. The sqlite write() therefore rejects duplicated primary-key column names up front with this ValueError.

Source

Thrown at python/pathway/io/sqlite/__init__.py:370

                "these columns in the Pathway table, or use "
                'output_table_type="snapshot".'
            )

    key_field_names: list[str] | None = None
    if primary_key is not None:
        # Duplicate entries in `primary_key` produce a nonsensical SQL
        # template — e.g. `PRIMARY KEY ("k", "k")` — which SQLite tolerates,
        # but the shared `SqlQueryTemplate` then reorders DELETE bindings
        # using the duplicated index and retractions silently match no
        # rows. Reject here with a clear message.
        names_seen: set[str] = set()
        duplicates: list[str] = []
        for pkey in primary_key:
            if pkey.name in names_seen and pkey.name not in duplicates:
                duplicates.append(pkey.name)
            names_seen.add(pkey.name)
        if duplicates:
            raise ValueError(
                f"primary_key contains duplicate column(s) {sorted(duplicates)}. "
                "Each column may appear at most once."
            )
        key_field_names = []
        for pkey in primary_key:
            # Raises ValueError when `pkey` belongs to a different table
            # or does not name a column of `table`, so users get a clear
            # message at write() time instead of an opaque runtime error.
            get_column_index(table, pkey)
            # Reject nullable primary-key columns. SQLite's
            # INTEGER PRIMARY KEY auto-assigns a rowid for NULL values,
            # which can silently collide with a later UPSERT and
            # overwrite unrelated rows; other PK types let NULLs through
            # but then DELETE ... WHERE pk = NULL never matches on
            # retractions. Neither case is what the user asked for.
            if isinstance(pkey._column.dtype, dt.Optional):
                raise ValueError(
                    f"primary_key column {pkey.name!r} is declared nullable "

View on GitHub (pinned to fa2f74a464)

Solutions

1. Remove the duplicate entry so each column appears exactly once: primary_key=[t.owner, t.pet].
2. If the key list is built dynamically, de-duplicate it while preserving order before calling write(), e.g. list(dict.fromkeys(names)).
3. Review whether the duplication came from merging two key specs (natural + surrogate) and drop the redundant one.

Example fix

# before
keys = [t.owner, t.pet, t.owner]
pw.io.sqlite.write(t, "db", "snap", output_table_type="snapshot", primary_key=keys)

# after
keys = [t.owner, t.pet]
pw.io.sqlite.write(t, "db", "snap", output_table_type="snapshot", primary_key=keys)

def dedupe_primary_key(primary_key):
    seen, unique = set(), []
    for col in primary_key:
        if col.name not in seen:
            seen.add(col.name)
            unique.append(col)
    if len(unique) != len(primary_key):
        raise ValueError("primary_key contains duplicate columns")
    return unique

When it happens

Trigger: Calling pw.io.sqlite.write(..., output_table_type="snapshot", primary_key=[t.k, t.k]) or any primary_key list where the same column reference (by name) appears more than once, e.g. built dynamically from a list with accidental repeats.

Common situations: Primary-key lists constructed programmatically from user input or config where the same key appears twice; concatenating key lists (natural key + surrogate key) that overlap; copy-paste of the same column reference into a composite key.

Related errors

• Incorrect S3 path: {s3_path}
• SchemaRegistryHeader.key must be a str, got {type(self.key).
• primary_key column {primary_key._name!r} does not belong to
• Defining a primary key in the schema is not supported for pw
• primary key field names must be specified for a snapshot mod

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/38489f32df9ab08c. Report an issue: GitHub.