{"record":{"id":"38b73aa9836f91c1","repo":"pathwaycom/pathway","slug":"the-sort-by-column-column-doesn-t-belong-to-the","errorCode":null,"errorMessage":"The sort_by column {column} doesn't belong to the table passed to pw.io.kafka.write.","messagePattern":"The sort_by column (.+?) doesn't belong to the table passed to pw\\.io\\.kafka\\.write\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/pathway/io/kafka/__init__.py","lineNumber":769,"sourceCode":"            sort_by=remapped_sort_by,\n        )\n    )\n\n\ndef _remap_sort_by(\n    sort_by: Iterable[ColumnReference] | None,\n    original_table: Table,\n    output_table: Table,\n) -> list[ColumnReference] | None:\n    if sort_by is None:\n        return None\n    remapped: list[ColumnReference] = []\n    for column in sort_by:\n        if column._table is output_table:\n            remapped.append(column)\n            continue\n        if column._table is not original_table:\n            raise ValueError(\n                f\"The sort_by column {column} doesn't belong to the table \"\n                \"passed to pw.io.kafka.write.\"\n            )\n        if column.name not in output_table._columns:\n            raise ValueError(\n                f\"The sort_by column {column.name!r} is not part of the \"\n                \"data being written. For 'raw' or 'plaintext' format, only \"\n                \"the 'value', 'key', 'topic_name' and 'headers' columns \"\n                \"are forwarded.\"\n            )\n        remapped.append(output_table[column.name])\n    return remapped\n\n\n__all__ = [\n    \"SchemaRegistryHeader\",\n    \"SchemaRegistrySettings\",\n    \"read\",","sourceCodeStart":751,"sourceCodeEnd":787,"githubUrl":"https://github.com/pathwaycom/pathway/blob/fa2f74a4649b7c5908690cf60137263d8d80de5f/python/pathway/io/kafka/__init__.py#L751-L787","documentation":"Pathway's Kafka writer can order rows within a batch via sort_by, but only using columns that belong either to the table passed to pw.io.kafka.write or to the internally built output table. This ValueError is raised when a sort_by ColumnReference comes from a different table (e.g. a filtered, joined, or retyped intermediate). Pathway rejects it early because it cannot map that column onto the data actually being written.","triggerScenarios":"Calling pw.io.kafka.write(table, ..., sort_by=[other_table.col]) where other_table is not the exact table object passed as the first argument (nor the internal output table). Typical with pw.Table.filter()/join() results: the connector receives the original table but sort_by references a column of a derived table.","commonSituations":"Building an output table, then passing an older table variable to write() while reusing column references captured from a later transform; mixing references captured before and after a with_columns() or filter() step.","solutions":["Pass sort_by references taken from the very same table object you pass as the first argument to pw.io.kafka.write","Assign the final transformed table to a variable (e.g. output = table.filter(...)) and use both output and output.col in the write call","If you need a column from another table, join/rename it into the written table first so all sort_by columns share one table"],"exampleFix":"// before\nfiltered = table.filter(t.value > 0)\npw.io.kafka.write(table, topic, sort_by=[filtered.ts])\n\n// after\nfiltered = table.filter(t.value > 0)\npw.io.kafka.write(filtered, topic, sort_by=[filtered.ts])","handlingStrategy":"validation","validationCode":"def check_sort_by(table, sort_by):\n    for c in sort_by:\n        assert c._table is table, f\"sort_by column {c} is not from the written table\"\n    return sort_by","typeGuard":"def columns_belong_to(table: pw.Table, cols: list[pw.ColumnReference]) -> bool:\n    return all(c._table is table for c in cols)","tryCatchPattern":"try:\n    pw.io.kafka.write(table, topic, sort_by=cols)\nexcept ValueError as e:\n    if \"doesn't belong to the table\" in str(e):\n        raise  # re-raise with context after logging table/col ids\n    raise","preventionTips":["Keep one variable for the final table and derive both the table argument and all sort_by references from it","Never pass column references captured before a transform alongside the post-transform table","Wrap connector setup in a small helper that asserts col._table is table for every column argument"],"tags":["kafka","pathway","sort","column-reference","validation"],"backgroundTag":null,"analyzedSha":"fa2f74a4649b7c5908690cf60137263d8d80de5f","analyzedAt":"2026-08-15T01:48:17.006Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}