pathwaycom/pathway · error · ValueError

The table has columns {sorted(conflicting)} which clash with

Error message

The table has columns {sorted(conflicting)} which clash with the reserved JSON fields written by the connector ('time', 'diff'). Rename or drop the conflicting column(s) before writing in 'json' format, otherwise the output JSON would contain duplicate keys.

What it means

Raised by MessageQueueOutputFormat.build when writing with format='json' and the table contains columns named 'time' or 'diff'. The json output format additionally writes Pathway's update metadata as top-level JSON fields 'time' and 'diff', so identically named table columns would produce duplicate keys in the emitted JSON object; the writer refuses rather than emit ambiguous payloads.

Source

Thrown at python/pathway/io/_utils.py:545

                        "produce a header with the same name. Alias one of "
                        "them to a different name (e.g. via `table.select(...)`) "
                        "to keep both as separate Kafka headers."
                    )
                header_fields[header.name] = cls.add_column_reference_to_extract(
                    header, columns_to_extract, extracted_field_indices
                )

        # Format-dependent parts: handle json and dsv separately
        if format == "json" or format == "dsv":
            if value is not None:
                raise ValueError(
                    f"'value' and format='{format}' cannot be set at the same time"
                )
            if format == "json":
                reserved = {"time", "diff"}
                conflicting = reserved.intersection(table._columns.keys())
                if conflicting:
                    raise ValueError(
                        f"The table has columns {sorted(conflicting)} which "
                        f"clash with the reserved JSON fields written by the "
                        f"connector ('time', 'diff'). Rename or drop the "
                        f"conflicting column(s) before writing in 'json' "
                        f"format, otherwise the output JSON would contain "
                        f"duplicate keys."
                    )
            for column_name in table._columns:
                cls.add_column_reference_to_extract(
                    table[column_name], columns_to_extract, extracted_field_indices
                )
            table = table.select(*columns_to_extract)
            data_format = api.DataFormat(
                format_type="jsonlines" if format == "json" else "dsv",
                key_field_names=[],
                value_fields=_format_output_value_fields(table),
                delimiter=delimiter,
                schema_registry_settings=maybe_schema_registry_settings(

View on GitHub (pinned to fa2f74a464)

Solutions

1. Rename the conflicting columns before writing: t = t.rename(event_time=pw.this.time).
2. Or drop them if not needed: t = t.without(pw.this.time, pw.this.diff).
3. Or choose format='dsv'/'raw' if the metadata fields are not required and renaming is unacceptable.

Example fix

# before
pw.io.kafka.write(t, ..., format='json')  # t has column 'time'

# after
t = t.rename(event_time=pw.this.time)
pw.io.kafka.write(t, ..., format='json')

conflicting = {'time', 'diff'} & set(table.column_names())
assert not conflicting, f"columns {sorted(conflicting)} clash with reserved json fields; rename or drop them"

def json_write_safe(table) -> bool:
    return not ({'time', 'diff'} & set(table.column_names()))

try:
    pw.io.kafka.write(t, ..., format='json')
except ValueError as e:
    if 'reserved JSON fields' in str(e):
        renames = {c: c + '_' for c in ('time', 'diff') if c in t.column_names()}
        pw.io.kafka.write(t.rename(**renames), ..., format='json')
    else:
        raise

When it happens

Trigger: pw.io.kafka.write(t, ..., format='json') where t already has columns named time or diff (e.g. an events table with a 'time' column, or a table that retains pathway's time/diff columns from a reducer or output-mode table).

Common situations: Domain models that legitimately use 'time' as a timestamp column name; tables derived from persistence/output formats that add time/diff columns.

Related errors

• {header.name!r} is reserved for the Pathway-injected headers
• Failed to install dependencies
• Column {pseudocolumn} has to contain integers only.
• Column {api.TIME_PSEUDOCOLUMN} cannot contain negative times
• Failed to detect the region of S3 bucket {bucket!r} (HTTP st

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/6463d2fd88884ca7. Report an issue: GitHub.