{"record":{"id":"907e633f8730ed63","repo":"pathwaycom/pathway","slug":"column-diff-pseudocolumn-can-only-contain-1-and","errorCode":null,"errorMessage":"Column {DIFF_PSEUDOCOLUMN} can only contain 1 and -1.","messagePattern":"Column (.+?) can only contain 1 and -1\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/pathway/internals/api.py","lineNumber":187,"sourceCode":"            for v in data[c]:\n                if v is not None:\n                    dtype = type(v)\n                    break\n            column_properties.append(ColumnProperties(dtype=dt.wrap(dtype).to_engine()))\n        connector_properties = ConnectorProperties(column_properties=column_properties)\n\n    assert len(connector_properties.column_properties) == len(\n        ordinary_columns\n    ), \"provided connector properties do not match the dataframe\"\n\n    input_data: CapturedStream = []\n    for i, index in enumerate(df.index):\n        key = ids[index]\n        values = [data[c][i] for c in ordinary_columns]\n        time = data[TIME_PSEUDOCOLUMN][i] if TIME_PSEUDOCOLUMN in data else 0\n        diff = data[DIFF_PSEUDOCOLUMN][i] if DIFF_PSEUDOCOLUMN in data else 1\n        if diff not in [-1, 1]:\n            raise ValueError(f\"Column {DIFF_PSEUDOCOLUMN} can only contain 1 and -1.\")\n        shard = data[SHARD_PSEUDOCOLUMN][i] if SHARD_PSEUDOCOLUMN in data else None\n        input_row = DataRow(\n            key, values, time=time, diff=diff, shard=shard, dtypes=dtypes\n        )\n        input_data.append(input_row)\n\n    return scope.static_table(input_data, connector_properties)\n\n\ndef squash_updates(\n    updates: CapturedStream, *, terminate_on_error: bool = True\n) -> CapturedTable:\n    state: CapturedTable = {}\n    updates.sort(key=lambda row: (row.time, row.diff))\n\n    def handle_error(row: DataRow, msg: str):\n        if terminate_on_error:\n            raise KeyError(msg)","sourceCodeStart":169,"sourceCodeEnd":205,"githubUrl":"https://github.com/pathwaycom/pathway/blob/fa2f74a4649b7c5908690cf60137263d8d80de5f/python/pathway/internals/api.py#L169-L205","documentation":"When a pandas DataFrame is converted into a Pathway table (via pw.debug.table_from_pandas / static_table_from_pandas), an optional '__diff__' pseudocolumn encodes row operations: 1 means insert and -1 means delete. Any other value (0, 2, booleans-as-strings, NaN-derived floats) is not a valid diff, so a ValueError is raised during conversion. Without this check the change stream fed into the engine would be malformed.","triggerScenarios":"pw.debug.table_from_pandas(df) where df contains a '__diff__' column holding 0, 2, or values like '1'/'-1' strings; computing __diff__ programmatically (e.g. 1 if added else 0) and forgetting that deletions must be -1; generating update streams from a diff tool that outputs 0 for unchanged rows.","commonSituations":"Replaying CDC/change streams captured from another system into Pathway for testing; building synthetic update dataframes in tests; filtering unchanged rows out but leaving their __diff__ value as 0.","solutions":["Map your diff values to the supported set: 1 for insert, -1 for delete.","Drop rows whose diff is 0 (unchanged) before conversion: df = df[df['__diff__'] != 0].","Rename or remove the column if it is not meant to be a diff column at all (the name '__diff__' is reserved)."],"exampleFix":"# before\ndf[\"__diff__\"] = df[\"change_type\"].map({\"insert\": 1, \"update\": 1, \"none\": 0})\ntable = pw.debug.table_from_pandas(df)\n\n# after\ndf[\"__diff__\"] = df[\"change_type\"].map({\"insert\": 1, \"delete\": -1, \"update\": 1})\ndf = df[df[\"__diff__\"].isin([1, -1])]\ntable = pw.debug.table_from_pandas(df)","handlingStrategy":"validation","validationCode":"DIFF = \"__diff__\"\nif DIFF in df.columns:\n    if not df[DIFF].isin([1, -1]).all():\n        bad = df.loc[~df[DIFF].isin([1, -1]), DIFF].unique().tolist()\n        raise ValueError(f\"__diff__ has unsupported values {bad}; allowed: 1, -1\")\n    df = df[df[DIFF].isin([1, -1])]\ntable = pw.debug.table_from_pandas(df)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Remember the reserved pseudocolumns __time__, __diff__, __shard__ when building replay dataframes.","Encode CDC updates as delete (-1) then insert (1), and drop unchanged (0) rows.","Unit-test replay dataframes: assert set(df['__diff__'].unique()) <= {1, -1}."],"tags":["pandas","testing","data-ingestion","validation"],"backgroundTag":null,"analyzedSha":"fa2f74a4649b7c5908690cf60137263d8d80de5f","analyzedAt":"2026-08-15T01:48:17.006Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}