pathwaycom/pathway · error · ValueError

Duplicate header name {header.name!r}: two columns produce a

Error message

Duplicate header name {header.name!r}: two columns produce a header with the same name. Alias one of them to a different name (e.g. via `table.select(...)`) to keep both as separate Kafka headers.

What it means

Raised by MessageQueueOutputFormat.build when two entries in the 'headers' list use the same column name. Each header becomes one Kafka header keyed by the column name, so two same-named headers would clobber each other; the duplicate is detected while building the header_fields mapping and rejected with a hint to alias.

Source

Thrown at python/pathway/io/_utils.py:525

            ):
                raise ValueError(
                    f"The key column must have one of the following types: {allowed_key_types}"
                )
            key_field_index = cls.add_column_reference_to_extract(
                key, columns_to_extract, extracted_field_indices
            )
        if headers is not None:
            reserved_header_names = {"pathway_time", "pathway_diff"}
            for header in headers:
                if header.name in reserved_header_names:
                    raise ValueError(
                        f"{header.name!r} is reserved for the Pathway-injected "
                        "headers (pathway_time / pathway_diff) and cannot be "
                        "used as a user header name. Alias the column to "
                        "another name with `table.select(<new_name>=...)`."
                    )
                if header.name in header_fields:
                    raise ValueError(
                        f"Duplicate header name {header.name!r}: two columns "
                        "produce a header with the same name. Alias one of "
                        "them to a different name (e.g. via `table.select(...)`) "
                        "to keep both as separate Kafka headers."
                    )
                header_fields[header.name] = cls.add_column_reference_to_extract(
                    header, columns_to_extract, extracted_field_indices
                )

        # Format-dependent parts: handle json and dsv separately
        if format == "json" or format == "dsv":
            if value is not None:
                raise ValueError(
                    f"'value' and format='{format}' cannot be set at the same time"
                )
            if format == "json":
                reserved = {"time", "diff"}
                conflicting = reserved.intersection(table._columns.keys())

View on GitHub (pinned to fa2f74a464)

Solutions

1. Alias one of the duplicates: t = t.rename(tag_source=pw.this.tag) and use distinct names in headers.
2. Deduplicate the headers list before the call: headers=list(dict.fromkeys(headers)).

Example fix

# before
pw.io.kafka.write(t, ..., headers=[pw.this.tag, pw.this.tag])

# after
headers = list(dict.fromkeys([pw.this.tag]))  # dedupe
pw.io.kafka.write(t, ..., headers=headers)

names = [h.name for h in headers]
assert len(names) == len(set(names)), f"duplicate header names: {sorted({n for n in names if names.count(n) > 1})}"

def no_duplicate_headers(headers) -> bool:
    names = [h.name for h in headers]
    return len(names) == len(set(names))

When it happens

Trigger: pw.io.kafka.write(t, ..., headers=[pw.this.tag, pw.this.tag]) — e.g. the same column listed twice, or two different columns that were both aliased to 'tag' in a select; programmatically building the headers list with a bug that appends a column twice.

Common situations: Header lists assembled from multiple sources (base columns + extra columns) that overlap; deduplication missing in generated configs.

Related errors

• {header.name!r} is reserved for the Pathway-injected headers
• Failed to install dependencies
• Column {pseudocolumn} has to contain integers only.
• Column {api.TIME_PSEUDOCOLUMN} cannot contain negative times
• Unsupported argument for {data_format!r} format: 'value'

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/e9c8abadba2d651a. Report an issue: GitHub.