pathwaycom/pathway · error · ValueError

Column {DIFF_PSEUDOCOLUMN} can only contain 1 and -1.

Error message

Column {DIFF_PSEUDOCOLUMN} can only contain 1 and -1.

What it means

When a pandas DataFrame is converted into a Pathway table (via pw.debug.table_from_pandas / static_table_from_pandas), an optional '__diff__' pseudocolumn encodes row operations: 1 means insert and -1 means delete. Any other value (0, 2, booleans-as-strings, NaN-derived floats) is not a valid diff, so a ValueError is raised during conversion. Without this check the change stream fed into the engine would be malformed.

Source

Thrown at python/pathway/internals/api.py:187

            for v in data[c]:
                if v is not None:
                    dtype = type(v)
                    break
            column_properties.append(ColumnProperties(dtype=dt.wrap(dtype).to_engine()))
        connector_properties = ConnectorProperties(column_properties=column_properties)

    assert len(connector_properties.column_properties) == len(
        ordinary_columns
    ), "provided connector properties do not match the dataframe"

    input_data: CapturedStream = []
    for i, index in enumerate(df.index):
        key = ids[index]
        values = [data[c][i] for c in ordinary_columns]
        time = data[TIME_PSEUDOCOLUMN][i] if TIME_PSEUDOCOLUMN in data else 0
        diff = data[DIFF_PSEUDOCOLUMN][i] if DIFF_PSEUDOCOLUMN in data else 1
        if diff not in [-1, 1]:
            raise ValueError(f"Column {DIFF_PSEUDOCOLUMN} can only contain 1 and -1.")
        shard = data[SHARD_PSEUDOCOLUMN][i] if SHARD_PSEUDOCOLUMN in data else None
        input_row = DataRow(
            key, values, time=time, diff=diff, shard=shard, dtypes=dtypes
        )
        input_data.append(input_row)

    return scope.static_table(input_data, connector_properties)


def squash_updates(
    updates: CapturedStream, *, terminate_on_error: bool = True
) -> CapturedTable:
    state: CapturedTable = {}
    updates.sort(key=lambda row: (row.time, row.diff))

    def handle_error(row: DataRow, msg: str):
        if terminate_on_error:
            raise KeyError(msg)

View on GitHub (pinned to fa2f74a464)

Solutions

1. Map your diff values to the supported set: 1 for insert, -1 for delete.
2. Drop rows whose diff is 0 (unchanged) before conversion: df = df[df['__diff__'] != 0].
3. Rename or remove the column if it is not meant to be a diff column at all (the name '__diff__' is reserved).

Example fix

# before
df["__diff__"] = df["change_type"].map({"insert": 1, "update": 1, "none": 0})
table = pw.debug.table_from_pandas(df)

# after
df["__diff__"] = df["change_type"].map({"insert": 1, "delete": -1, "update": 1})
df = df[df["__diff__"].isin([1, -1])]
table = pw.debug.table_from_pandas(df)

DIFF = "__diff__"
if DIFF in df.columns:
    if not df[DIFF].isin([1, -1]).all():
        bad = df.loc[~df[DIFF].isin([1, -1]), DIFF].unique().tolist()
        raise ValueError(f"__diff__ has unsupported values {bad}; allowed: 1, -1")
    df = df[df[DIFF].isin([1, -1])]
table = pw.debug.table_from_pandas(df)

When it happens

Trigger: pw.debug.table_from_pandas(df) where df contains a '__diff__' column holding 0, 2, or values like '1'/'-1' strings; computing __diff__ programmatically (e.g. 1 if added else 0) and forgetting that deletions must be -1; generating update streams from a diff tool that outputs 0 for unchanged rows.

Common situations: Replaying CDC/change streams captured from another system into Pathway for testing; building synthetic update dataframes in tests; filtering unchanged rows out but leaving their __diff__ value as 0.

Related errors

• duplicated entries for key {row.key}
• string {freq} cannot be parsed as a duration
• schema does not have columns {missing_columns}
• there are extra columns: {extra_columns} which are not prese
• Failed to install dependencies

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/907e633f8730ed63. Report an issue: GitHub.