pathwaycom/pathway · error · ValueError

_external_diff_column is only supported for the snapshot tab

Error message

_external_diff_column is only supported for the snapshot table type

What it means

Raised by pw.io.postgres.write() when _external_diff_column is passed while output_table_type is not "snapshot". The external diff column (which exposes the +1/-1 change flag to the pipeline) is only implemented for the snapshot writer; in stream mode diff information is already persisted as the appended diff column.

Source

Thrown at python/pathway/io/postgres/__init__.py:877

                "stream_of_changes mode. Rename the column(s) in your "
                "schema or switch to output_table_type='snapshot' "
                "which does not append these metadata columns."
            )

    data_storage = api.DataStorage(
        storage_type="postgres",
        connection_string=_connection_string_from_settings(postgres_settings),
        max_batch_size=max_batch_size,
        table_name=table_name,
        schema_name=schema_name,
        table_writer_init_mode=init_mode_from_str(init_mode),
        snapshot_maintenance_on_output=is_snapshot_mode,
        tls_settings=tls.settings,
    )

    if not is_snapshot_mode:
        if _external_diff_column is not None:
            raise ValueError(
                "_external_diff_column is only supported for the snapshot table type"
            )
        if primary_key is not None:
            raise ValueError(
                "primary_key can only be specified for the snapshot table type"
            )
    else:
        # Snapshot mode requires at least one primary-key column —
        # the writer's INSERT ... ON CONFLICT (...) DO UPDATE
        # statement is malformed without one. If we let an empty list
        # reach the engine it would error out only AFTER ``init_mode``
        # has already mutated the destination (CREATE TABLE for
        # ``"replace"`` / ``"create_if_not_exists"``), and under
        # multi-worker (PATHWAY_THREADS > 1) the worker that loses the
        # CREATE race observes the partially-created table and
        # surfaces a less specific error instead — making any
        # message-based test flaky. Reject at call time so no DB side
        # effect happens.

View on GitHub (pinned to fa2f74a464)

Solutions

1. Add output_table_type="snapshot" and a primary_key=[...] to the write call.
2. If you need the raw diff in stream semantics, read the appended diff column from the destination table instead of using _external_diff_column.

Example fix

# before
pw.io.postgres.write(t, parts, "tbl", _external_diff_column=t.d)
# after
pw.io.postgres.write(t, parts, "tbl", output_table_type="snapshot", primary_key=[t.k], _external_diff_column=t.d)

if _external_diff_column is not None:
    assert output_table_type == "snapshot", "_external_diff_column requires snapshot mode"

When it happens

Trigger: pw.io.postgres.write(table, parts, "tbl", _external_diff_column=table.d) with default output_table_type (stream_of_changes).

Common situations: Advanced users wiring the change flag into the dataflow and forgetting the snapshot requirement; internal/experimental API usage copied between connectors.

Related errors

• primary_key can only be specified for the snapshot table typ
• primary key field names must be specified for a snapshot mod
• _external_diff_column can only have an integer type
• primary_key references column(s) {foreign} that are not pres
• primary_key column '{pkey_field.name}' is declared nullable;

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/ec705a427b721ef8. Report an issue: GitHub.