pathwaycom/pathway · error · ValueError

Pathway schema column(s) {offending} collide with the 'time'

Error message

Pathway schema column(s) {offending} collide with the 'time' and 'diff' metadata columns appended in stream_of_changes mode. Rename the column(s) in your schema or switch to output_table_type='snapshot' which does not append these metadata columns.

What it means

In stream-of-changes mode, the MySQL sink appends 'time' and 'diff' metadata columns to the destination table; a schema column with the same name (case-insensitively) would make the engine fail mid-run with an opaque 'Duplicate column name' error. pw.io.mysql.write raises this ValueError at call time naming the colliding columns. Snapshot mode does not append these columns, so the check only applies in stream mode.

Source

Thrown at python/pathway/io/mysql/__init__.py:402

    is_snapshot_mode = output_table_type == SNAPSHOT_OUTPUT_TABLE_TYPE

    # Stream-of-changes mode appends ``time BIGINT NOT NULL, diff SMALLINT
    # NOT NULL`` metadata columns to the generated CREATE TABLE / INSERT. A
    # user column with one of those names (case-insensitive -- MySQL column
    # names are not case-sensitive) would otherwise be emitted twice and
    # the engine worker would fail mid-run with an opaque "Duplicate column
    # name" error. Snapshot mode does not append these columns, so the
    # check only applies in stream mode.
    if not is_snapshot_mode:
        offending = sorted(
            {
                column_name
                for column_name in table.schema.column_names()
                if column_name.lower() in ("time", "diff")
            }
        )
        if offending:
            raise ValueError(
                f"Pathway schema column(s) {offending} collide with "
                "the 'time' and 'diff' metadata columns appended in "
                "stream_of_changes mode. Rename the column(s) in your "
                "schema or switch to output_table_type='snapshot' "
                "which does not append these metadata columns."
            )

    data_storage = api.DataStorage(
        storage_type="mysql",
        connection_string=connection_string,
        max_batch_size=max_batch_size,
        table_name=table_name,
        table_writer_init_mode=init_mode_from_str(init_mode),
        snapshot_maintenance_on_output=is_snapshot_mode,
    )

    key_field_names = None
    if primary_key is not None:

View on GitHub (pinned to fa2f74a464)

Solutions

Rename the colliding column(s) before the write, e.g. time -> event_time via table.select() or table.rename().
Or pass output_table_type="snapshot" to skip appending metadata columns.
Adopt a naming convention that avoids bare 'time'/'diff' in sink-bound tables.

Example fix

# before
table = table.select(table.id, table.time, table.amount)
pw.io.mysql.write(table, "payments")

# after
table = table.select(table.id, event_time=table.time, table.amount)
pw.io.mysql.write(table, "payments")

Defensive patterns

Strategy: validation

Validate before calling

RESERVED = {"time", "diff"}

bad = sorted(n for n in table.schema.column_names() if n.lower() in RESERVED)
if bad and output_table_type != "snapshot":
    table = table.select(**{
        **{c: pw.this[c] for c in table.schema.column_names()},
        **{c: pw.this[c] for c in bad},
    }).rename(**{c: f"src_{c}" for c in bad})

Type guard

def is_safe_for_stream_sink(column_names: list[str]) -> bool:
    return all(n.lower() not in ("time", "diff") for n in column_names)

Try / catch

try:
    pw.io.mysql.write(table, "events")
except ValueError as e:
    if "metadata columns" in str(e):
        pw.io.mysql.write(table, "events", output_table_type="snapshot")
    else:
        raise

Prevention

Rename 'time'/'diff' columns (event_time, change_diff) before MySQL stream sinks.
Centralize final-projection selects before sinks so renaming is one obvious place.
Snapshot mode avoids the reserved columns entirely if renaming is not an option.

When it happens

Trigger: Calling pw.io.mysql.write(table, table_name) with the default output_table_type while the table schema contains a column named time, Time, diff, or DIFF.

Common situations: Streaming event tables with a 'time' timestamp column; audit/change tables with a 'diff' column; porting pipelines from other sinks where these names are allowed.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/4feca197d9e4bf45. Report an issue: GitHub.