pathwaycom/pathway · error · ValueError

The sort_by column {column.name!r} is not part of the data b

Error message

The sort_by column {column.name!r} is not part of the data being written. For 'raw' or 'plaintext' format, only the 'value', 'key', 'topic_name' and 'headers' columns are forwarded.

What it means

When pw.io.kafka.write uses the 'raw' or 'plaintext' format, only the 'value', 'key', 'topic_name' and 'headers' columns survive into the output table. If sort_by names any other column of the input table, Pathway cannot order the messages by it because that column is not part of the payload being written, so it raises this ValueError.

Source

Thrown at python/pathway/io/kafka/__init__.py:774

def _remap_sort_by(
    sort_by: Iterable[ColumnReference] | None,
    original_table: Table,
    output_table: Table,
) -> list[ColumnReference] | None:
    if sort_by is None:
        return None
    remapped: list[ColumnReference] = []
    for column in sort_by:
        if column._table is output_table:
            remapped.append(column)
            continue
        if column._table is not original_table:
            raise ValueError(
                f"The sort_by column {column} doesn't belong to the table "
                "passed to pw.io.kafka.write."
            )
        if column.name not in output_table._columns:
            raise ValueError(
                f"The sort_by column {column.name!r} is not part of the "
                "data being written. For 'raw' or 'plaintext' format, only "
                "the 'value', 'key', 'topic_name' and 'headers' columns "
                "are forwarded."
            )
        remapped.append(output_table[column.name])
    return remapped


__all__ = [
    "SchemaRegistryHeader",
    "SchemaRegistrySettings",
    "read",
    "write",
]

View on GitHub (pinned to fa2f74a464)

Solutions

1. Use format='json' (or another format that forwards all columns) if you need to sort by arbitrary columns
2. Restrict sort_by to the forwarded columns: value, key, topic_name, headers
3. Encode the sort key into the value/key column before writing so ordering can use it

Example fix

# before
pw.io.kafka.write(table, topic, format='raw', value=table.payload, sort_by=[table.created_at])

# after
pw.io.kafka.write(table, topic, format='json', sort_by=[table.created_at])

FORWARDED = {"value", "key", "topic_name", "headers"}
if format in ("raw", "plaintext"):
    assert all(c._name in FORWARDED for c in sort_by), "sort_by must use forwarded columns for raw/plaintext"

def sort_columns_forwarded(fmt: str, cols: list[pw.ColumnReference]) -> bool:
    return fmt not in ("raw", "plaintext") or all(c._name in {"value", "key", "topic_name", "headers"} for c in cols)

try:
    pw.io.kafka.write(table, topic, format="raw", value=table.payload, sort_by=cols)
except ValueError as e:
    if "not part of the data being written" in str(e):
        cols = [c for c in cols if c._name in {"value", "key", "topic_name", "headers"}]
    else:
        raise

When it happens

Trigger: pw.io.kafka.write(table, topic, format='raw'|'plaintext', sort_by=[table.some_column]) where some_column is not one of value/key/topic_name/headers.

Common situations: Reusing a sort_by list that worked with format='json' (which forwards all columns) after switching to 'raw'; sorting by a timestamp or id column that exists only on the input table.

Related errors

• Unsupported argument for {data_format!r} format: 'value'
• 'schema_registry_settings' is only meaningful for the 'json'
• 'value' and format='{format}' cannot be set at the same time
• Unsupported format: {format}
• The sort_by column {column} doesn't belong to the table pass

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/45d255f6745834de. Report an issue: GitHub.