pathwaycom/pathway · error · ValueError

Duplicate header name

Error message

Duplicate header name {header.name!r}: two columns produce a header with the same name. Alias one of them to a different name (e.g. via `table.select(...)`) to keep both as separate Kafka headers.

What it means

Raised by MessageQueueOutputFormat.build when two entries in the 'headers' list use the same column name. Each header becomes one Kafka header keyed by the column name, so two same-named headers would clobber each other; the duplicate is detected while building the header_fields mapping and rejected with a hint to alias.

Solutions

Alias one of the duplicates: t = t.rename(tag_source=pw.this.tag) and use distinct names in headers.
Deduplicate the headers list before the call: headers=list(dict.fromkeys(headers)).

Example fix

# before
pw.io.kafka.write(t, ..., headers=[pw.this.tag, pw.this.tag])

# after
headers = list(dict.fromkeys([pw.this.tag]))  # dedupe
pw.io.kafka.write(t, ..., headers=headers)

Defensive patterns

Strategy: validation

Validate before calling

names = [h.name for h in headers]
assert len(names) == len(set(names)), f"duplicate header names: {sorted({n for n in names if names.count(n) > 1})}"

Type guard

def no_duplicate_headers(headers) -> bool:
    names = [h.name for h in headers]
    return len(names) == len(set(names))

Prevention

Deduplicate header lists: headers = list(dict.fromkeys(headers)).
When merging header lists from multiple sources, merge by name and detect conflicts.

When it happens

Trigger: pw.io.kafka.write(t, ..., headers=[pw.this.tag, pw.this.tag]) — e.g. the same column listed twice, or two different columns that were both aliased to 'tag' in a select; programmatically building the headers list with a bug that appends a column twice.

Common situations: Header lists assembled from multiple sources (base columns + extra columns) that overlap; deduplication missing in generated configs.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/e9c8abadba2d651a. Report an issue: GitHub.

Appendix: source

Thrown at python/pathway/io/_utils.py:525

            ):
                raise ValueError(
                    f"The key column must have one of the following types: {allowed_key_types}"
                )
            key_field_index = cls.add_column_reference_to_extract(
                key, columns_to_extract, extracted_field_indices
            )
        if headers is not None:
            reserved_header_names = {"pathway_time", "pathway_diff"}
            for header in headers:
                if header.name in reserved_header_names:
                    raise ValueError(
                        f"{header.name!r} is reserved for the Pathway-injected "
                        "headers (pathway_time / pathway_diff) and cannot be "
                        "used as a user header name. Alias the column to "
                        "another name with `table.select(<new_name>=...)`."
                    )
                if header.name in header_fields:
                    raise ValueError(
                        f"Duplicate header name {header.name!r}: two columns "
                        "produce a header with the same name. Alias one of "
                        "them to a different name (e.g. via `table.select(...)`) "
                        "to keep both as separate Kafka headers."
                    )
                header_fields[header.name] = cls.add_column_reference_to_extract(
                    header, columns_to_extract, extracted_field_indices
                )

        # Format-dependent parts: handle json and dsv separately
        if format == "json" or format == "dsv":
            if value is not None:
                raise ValueError(
                    f"'value' and format='{format}' cannot be set at the same time"
                )
            if format == "json":
                reserved = {"time", "diff"}
                conflicting = reserved.intersection(table._columns.keys())

View on GitHub (pinned to fa2f74a464)