{"record":{"id":"45d255f6745834de","repo":"pathwaycom/pathway","slug":"the-sort-by-column-column-name-r-is-not-part-of","errorCode":null,"errorMessage":"The sort_by column {column.name!r} is not part of the data being written. For 'raw' or 'plaintext' format, only the 'value', 'key', 'topic_name' and 'headers' columns are forwarded.","messagePattern":"The sort_by column (.+?) is not part of the data being written\\. For 'raw' or 'plaintext' format, only the 'value', 'key', 'topic_name' and 'headers' columns are forwarded\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/pathway/io/kafka/__init__.py","lineNumber":774,"sourceCode":"def _remap_sort_by(\n    sort_by: Iterable[ColumnReference] | None,\n    original_table: Table,\n    output_table: Table,\n) -> list[ColumnReference] | None:\n    if sort_by is None:\n        return None\n    remapped: list[ColumnReference] = []\n    for column in sort_by:\n        if column._table is output_table:\n            remapped.append(column)\n            continue\n        if column._table is not original_table:\n            raise ValueError(\n                f\"The sort_by column {column} doesn't belong to the table \"\n                \"passed to pw.io.kafka.write.\"\n            )\n        if column.name not in output_table._columns:\n            raise ValueError(\n                f\"The sort_by column {column.name!r} is not part of the \"\n                \"data being written. For 'raw' or 'plaintext' format, only \"\n                \"the 'value', 'key', 'topic_name' and 'headers' columns \"\n                \"are forwarded.\"\n            )\n        remapped.append(output_table[column.name])\n    return remapped\n\n\n__all__ = [\n    \"SchemaRegistryHeader\",\n    \"SchemaRegistrySettings\",\n    \"read\",\n    \"write\",\n]\n","sourceCodeStart":756,"sourceCodeEnd":790,"githubUrl":"https://github.com/pathwaycom/pathway/blob/fa2f74a4649b7c5908690cf60137263d8d80de5f/python/pathway/io/kafka/__init__.py#L756-L790","documentation":"When pw.io.kafka.write uses the 'raw' or 'plaintext' format, only the 'value', 'key', 'topic_name' and 'headers' columns survive into the output table. If sort_by names any other column of the input table, Pathway cannot order the messages by it because that column is not part of the payload being written, so it raises this ValueError.","triggerScenarios":"pw.io.kafka.write(table, topic, format='raw'|'plaintext', sort_by=[table.some_column]) where some_column is not one of value/key/topic_name/headers.","commonSituations":"Reusing a sort_by list that worked with format='json' (which forwards all columns) after switching to 'raw'; sorting by a timestamp or id column that exists only on the input table.","solutions":["Use format='json' (or another format that forwards all columns) if you need to sort by arbitrary columns","Restrict sort_by to the forwarded columns: value, key, topic_name, headers","Encode the sort key into the value/key column before writing so ordering can use it"],"exampleFix":"# before\npw.io.kafka.write(table, topic, format='raw', value=table.payload, sort_by=[table.created_at])\n\n# after\npw.io.kafka.write(table, topic, format='json', sort_by=[table.created_at])","handlingStrategy":"validation","validationCode":"FORWARDED = {\"value\", \"key\", \"topic_name\", \"headers\"}\nif format in (\"raw\", \"plaintext\"):\n    assert all(c._name in FORWARDED for c in sort_by), \"sort_by must use forwarded columns for raw/plaintext\"","typeGuard":"def sort_columns_forwarded(fmt: str, cols: list[pw.ColumnReference]) -> bool:\n    return fmt not in (\"raw\", \"plaintext\") or all(c._name in {\"value\", \"key\", \"topic_name\", \"headers\"} for c in cols)","tryCatchPattern":"try:\n    pw.io.kafka.write(table, topic, format=\"raw\", value=table.payload, sort_by=cols)\nexcept ValueError as e:\n    if \"not part of the data being written\" in str(e):\n        cols = [c for c in cols if c._name in {\"value\", \"key\", \"topic_name\", \"headers\"}]\n    else:\n        raise","preventionTips":["When using raw/plaintext format, restrict sort_by to value/key/topic_name/headers","When switching formats, re-audit the sort_by list","Encode ordering keys into the value column if you must keep raw format"],"tags":["kafka","pathway","sort","format","validation"],"backgroundTag":null,"analyzedSha":"fa2f74a4649b7c5908690cf60137263d8d80de5f","analyzedAt":"2026-08-15T01:48:17.006Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}