pathwaycom/pathway · error · ValueError

only diffs of 1 and -1 are supported

Error message

only diffs of 1 and -1 are supported

What it means

A single pw.io.airbyte.read call can wrap multiple streams, but they must all share the same sync_mode because Pathway's persistence/state handling is uniform per connector. The catalog check raises ValueError when it detects a mix (e.g. one incremental, one full_refresh).

Source

Thrown at python/pathway/debug/__init__.py:573

            warn(
                "timestamps are required to be even; all timestamps will be doubled",
                stacklevel=_stacklevel + 1,
            )
            batches = {2 * timestamp: batches[timestamp] for timestamp in batches}

        for timestamp in sorted(batches):
            self._advance_time_for_all_workers(unique_name, workers, timestamp)
            batch = batches[timestamp]
            for worker, changes in batch.items():
                for diff, key, values in changes:
                    if diff == 1:
                        event = api.SnapshotEvent.insert(key, values)
                        self.events[(unique_name, worker)] += [event] * diff
                    elif diff == -1:
                        event = api.SnapshotEvent.delete(key, values)
                        self.events[(unique_name, worker)] += [event] * (-diff)
                    else:
                        raise ValueError("only diffs of 1 and -1 are supported")

        return read(
            _EmptyConnectorSubject(datasource_name="debug.stream-generator"),
            name=unique_name,
            schema=schema,
        )

    def table_from_list_of_batches_by_workers(
        self,
        batches: list[dict[int, list[dict[str, api.Value]]]],
        schema: type[Schema],
        _stacklevel: int = 1,
    ) -> Table:
        """A function that creates a table from a list of batches, where each batch is a mapping
        from worker id to a list of rows processed by this worker in this batch.
        Each row is a mapping from column name to a value.

        Args:

View on GitHub (pinned to fa2f74a464)

Solutions

1. Make all streams use the same sync_mode (usually "incremental").
2. If modes genuinely must differ, split into two pw.io.airbyte.read calls — one per sync mode — and union the resulting tables.
3. Regenerate the streams list programmatically so every entry gets the same mode string.

Example fix

# before
streams=[{"stream":"a","sync_mode":"incremental"}, {"stream":"b","sync_mode":"full_refresh"}]
t = pw.io.airbyte.read(..., streams=streams)

# after
t_a = pw.io.airbyte.read(..., streams=[{"stream":"a","sync_mode":"incremental"}])
t_b = pw.io.airbyte.read(..., streams=[{"stream":"b","sync_mode":"incremental"}])

modes = {s["sync_mode"] for s in streams if isinstance(s, dict)}
if len(modes) > 1:
    raise ValueError(f"mixed sync_modes {modes}; split into separate read calls")
pw.io.airbyte.read(..., streams=streams)

When it happens

Trigger: Passing a streams list where at least two entries have different sync_mode values, e.g. [{"stream":"a","sync_mode":"incremental"},{"stream":"b","sync_mode":"full_refresh"}].

Common situations: User replicates a multi-stream Airbyte source whose default catalog mixes modes; or merges streams from different sources into one read call during a migration.

Related errors

• negative timestamp cannot be used
• demo.noisy_linear_stream error: nb_rows should be strictly p
• demo.range_stream error: nb_rows should be strictly positive
• demo.replay_csv_with_time: unit should be either 's', 'ms, '
• Failed to install dependencies

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/05647bd2da4f868a. Report an issue: GitHub.