pathwaycom/pathway · error · ValueError

_external_diff_column can only have an integer type

Error message

_external_diff_column can only have an integer type

What it means

Raised by pw.io.postgres.write() when _external_diff_column is provided but its dtype is not Pathway's INT. The engine side expects a native integer (+1/-1) for the change flag, so float, string, or pointer-typed columns are rejected at call time.

Source

Thrown at python/pathway/io/postgres/__init__.py:904

        # the writer's INSERT ... ON CONFLICT (...) DO UPDATE
        # statement is malformed without one. If we let an empty list
        # reach the engine it would error out only AFTER ``init_mode``
        # has already mutated the destination (CREATE TABLE for
        # ``"replace"`` / ``"create_if_not_exists"``), and under
        # multi-worker (PATHWAY_THREADS > 1) the worker that loses the
        # CREATE race observes the partially-created table and
        # surfaces a less specific error instead — making any
        # message-based test flaky. Reject at call time so no DB side
        # effect happens.
        if primary_key is None or len(primary_key) == 0:
            raise ValueError(
                "primary key field names must be specified for a snapshot mode"
            )
    if (
        _external_diff_column is not None
        and _external_diff_column._column.dtype != dtype.INT
    ):
        raise ValueError("_external_diff_column can only have an integer type")

    external_diff_column_index = get_column_index(table, _external_diff_column)
    key_field_names = None
    if primary_key is not None:
        key_field_names = [pkey_field.name for pkey_field in primary_key]
        # `primary_key=[other_table.col]` (or a reference whose name
        # simply isn't in `table`) is accepted today but then either
        # generates a malformed CREATE TABLE (``PRIMARY KEY
        # ("unknown")``) on init or produces an UPSERT that panics at
        # flush. Reject up-front with a clear message.
        table_columns = set(table.schema.column_names())
        foreign = sorted(n for n in key_field_names if n not in table_columns)
        if foreign:
            raise ValueError(
                f"primary_key references column(s) {foreign} that are "
                "not present in the written table; pass ColumnReferences "
                "from the table being written, e.g. primary_key=[table.k]."
            )

View on GitHub (pinned to fa2f74a464)

Solutions

1. Cast the column to int before the write: t.with_columns(d=pw.this.d.astype(int)).
2. Or produce it as int from the start (annotate the schema field or the UDF return type as int).

Example fix

# before
@pw.udf
def diff(x: float) -> float:
    return 1.0 if x > 0 else -1.0
# after
@pw.udf
def diff(x: float) -> int:
    return 1 if x > 0 else -1

import pathway as pw
if _external_diff_column is not None:
    assert _external_diff_column._column.dtype == pw.dtype.INT, "diff column must be INT"

def is_int_column(col) -> bool:
    import pathway as pw
    return col is not None and col._column.dtype == pw.dtype.INT

When it happens

Trigger: pw.io.postgres.write(t, parts, "tbl", output_table_type="snapshot", primary_key=[t.k], _external_diff_column=t.d) where t.d is float, str, or bool typed in the schema.

Common situations: Diff columns produced by UDFs that return float(1)/float(-1) or Python objects; schemas inferred from pandas/Arrow where integers become int64-but-nullable or float columns.

Related errors

• primary key field names must be specified for a snapshot mod
• Incorrect S3 path: {s3_path}
• The topic name column must have a string type, however {topi
• The key column must have one of the following types: {allowe
• The value column must have one of the following types: {allo

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/9b6151a0cc39f418. Report an issue: GitHub.