pathwaycom/pathway · error · ValueError

Unable to process negative update with this stateful reducer

Error message

Unable to process negative update with this stateful reducer.

What it means

Pathway wraps stateful custom reducers (classes with update/compute_result) for use in reduce() over tables with changing data. When the accumulator has no neutral() and no retract() implementation, it cannot represent or process row deletions. This instance fires when the state is still None (nothing accumulated yet) and the update batch contains only negative (retraction) rows — there is no way to apply a deletion to an empty/uninitialized state.

Source

Thrown at python/pathway/internals/custom_reducers.py:365

                    positive_updates.extend(_positive_updates)
                    _positive_updates = []
                    state = None
                acc = Counter(positive_updates)
                acc.subtract(negative_updates)
                assert all(x >= 0 for x in acc.values())
                positive_updates = list(acc.elements())
                negative_updates = []

            if state is None:
                if neutral_available:
                    state = reducer_cls.neutral()
                    _positive_updates = []
                    _cnt = 0
                elif len(positive_updates) == 0:
                    if len(negative_updates) == 0:
                        return None
                    else:
                        raise ValueError(
                            "Unable to process negative update with this stateful reducer."
                        )
                else:
                    if sort_by_available:
                        positive_updates.sort(
                            key=lambda x: reducer_cls.sort_by(list(x))
                        )
                    state = reducer_cls.from_row(list(positive_updates[0]))
                    if not retract_available:
                        _positive_updates = positive_updates[0:1]
                        _cnt = 0
                    else:
                        _positive_updates = []
                        _cnt = 1
                    positive_updates = positive_updates[1:]

            updates = [(row_up, False) for row_up in positive_updates] + [
                (row_up, True) for row_up in negative_updates

View on GitHub (pinned to fa2f74a464)

Solutions

1. Implement retract(self, other) on the accumulator class so deletions can be undone.
2. Alternatively implement neutral() so an identity state exists to retract from.
3. If the input is genuinely append-only, ensure it is declared/ingested as append-only so Pathway does not deliver negative updates.
4. Prefer a built-in reducer (sum, count, min...) which supports retractions when your aggregation permits it.

Example fix

# before
class MySum(StatefulReducer):
    def __init__(self): self.total = 0
    def update(self, other): self.total += other
    def compute_result(self): return self.total

# after
class MySum(StatefulReducer):
    def __init__(self): self.total = 0
    def update(self, other): self.total += other
    def retract(self, other): self.total -= other
    def compute_result(self): return self.total

assert hasattr(ReducerCls, 'retract') or hasattr(ReducerCls, 'neutral'), 'custom reducer must support retractions (retract/neutral) on mutable sources'

def reducer_handles_retractions(cls) -> bool:
    return any('retract' in vars(c) or 'neutral' in vars(c) for c in cls.__mro__ if c is not object)

When it happens

Trigger: Using a custom reducer via pw.reducers.udf_reducer(CustomCls) where CustomCls defines neither neutral() nor retract(), applied to a non-append-only input source (e.g. Kafka with updates/deletes, mutable CSVs) whose first batch for some group is purely retractions.

Common situations: Custom reducers tested only on static/debug tables (append-only) then deployed against streaming sources that emit deletions; windowed reductions over tables where forgetting produces retraction batches.

Related errors

• Failed to install dependencies
• Column {pseudocolumn} has to contain integers only.
• Column {api.TIME_PSEUDOCOLUMN} cannot contain negative times
• Column {api.DIFF_PSEUDOCOLUMN} can only have 1 and -1 values
• parameters `schema` and `id_from` are mutually exclusive

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/6fc55ac94dad62b7. Report an issue: GitHub.