pathwaycom/pathway · error · ValueError

demo.replay_csv_with_time: unit should be either 's', 'ms, '

Error message

demo.replay_csv_with_time: unit should be either 's', 'ms, 'us', or 'ns'.

What it means

Airbyte connector state persistence maps one Pathway persistence subject to one Airbyte state blob, which is only well-defined for a single stream. on_persisted_run raises RuntimeError when a persisted run configures more than one stream.

Source

Thrown at python/pathway/demo/__init__.py:293

        autocommit_duration_ms: the maximum time between two commits. Every
          autocommit_duration_ms milliseconds, the updates received by the connector are
          committed and pushed into Pathway Live Data Framework's computation graph.
        speedup: Produce stream `speedup` times faster than it would result from the time column.

    Returns:
        Table: The table read.

    Note: the CSV files should follow a standard CSV settings. The separator is ',', the
    quotechar is '"', and there is no escape.

    """

    time_column_type = schema.typehints().get(time_column, None)
    if time_column_type != int and time_column_type != float:
        raise ValueError("Invalid schema. Time columns must be int or float.")

    if unit not in ["s", "ms", "us", "ns"]:
        raise ValueError(
            "demo.replay_csv_with_time: unit should be either 's', 'ms, 'us', or 'ns'."
        )

    unit_factor = 1
    match unit:
        case "ms":
            unit_factor = 1000
        case "us":
            unit_factor = 1_000_000
        case "ns":
            unit_factor = 1_000_000_000
        case _:
            unit_factor = 1
    speedup *= unit_factor

    columns = set(schema.column_names())

    class FileStreamSubject(pw.io.python.ConnectorSubject):

View on GitHub (pinned to fa2f74a464)

Solutions

1. Split into one pw.io.airbyte.read call per stream, each with a unique name and its own persistence state.
2. Union the resulting tables afterwards if a single table view is needed.
3. Alternatively drop persistence for this connector if multi-stream resume is not required.

Example fix

# before
t = pw.io.airbyte.read(..., streams=["users","orders"], persistence_mode="syncing", persistence_config=cfg)

# after
t_users = pw.io.airbyte.read(..., streams=["users"], name="airbyte_users", persistence_mode="syncing", persistence_config=cfg)
t_orders = pw.io.airbyte.read(..., streams=["orders"], name="airbyte_orders", persistence_mode="syncing", persistence_config=cfg)

if persistence_enabled and isinstance(streams, list) and len(streams) > 1:
    raise ValueError(
        "persistent airbyte connectors support one stream each; split the read"
    )
pw.io.airbyte.read(..., streams=streams)

When it happens

Trigger: Running pw.io.airbyte.read with persistence enabled (pw.persistence.Config / pw.run with persistence) while the streams argument lists two or more streams.

Common situations: A working multi-stream non-persistent setup is later migrated to persistent mode for exactly-once resume; the same configuration now fails on restart setup.

Related errors

• only diffs of 1 and -1 are supported
• Invalid schema. Time columns must be int or float.
• Column {api.DIFF_PSEUDOCOLUMN} can only have 1 and -1 values
• negative timestamp cannot be used
• demo.generate_custom_stream error: nb_rows should be None or

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/4b608ecb3790daac. Report an issue: GitHub.