{"record":{"id":"66efa76c8d145e68","repo":"pathwaycom/pathway","slug":"the-field-column-name-in-the-schema-of-the-sou","errorCode":null,"errorMessage":"The field '{column_name}' in the schema of the source '{datasource.name}' has unsupported source component: '{column_data.source_component}'","messagePattern":"The field '(.+?)' in the schema of the source '(.+?)' has unsupported source component: '(.+?)'","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/pathway/internals/table_io.py","lineNumber":56,"sourceCode":"    *,\n    supported_components: Iterable[str] = ...,\n) -> TTable: ...\n\n\ndef table_from_datasource(\n    datasource: datasources.DataSource,\n    debug_datasource: datasources.StaticDataSource | None = None,\n    table_cls: type[tables.Table] = tables.Table,\n    *,\n    supported_components: Iterable[str] = (PAYLOAD_SOURCE_COMPONENT,),\n) -> tables.Table:\n    for column_name, column_data in datasource.schema.columns().items():\n        if column_data.source_component not in supported_components:\n            error_message = (\n                f\"The field '{column_name}' in the schema of the source '{datasource.name}' \"\n                f\"has unsupported source component: '{column_data.source_component}'\"\n            )\n            raise ValueError(error_message)\n\n    return parse_graphs.G.add_operator(\n        lambda id: operators.InputOperator(datasource, id, debug_datasource),\n        lambda operator: operator(table_cls),\n    )\n\n\ndef table_to_datasink(\n    table: tables.Table, datasink: datasinks.DataSink, *, special: bool = False\n) -> operators.OutputOperator:\n    datasink.check_sort_by_columns(table)\n    return parse_graphs.G.add_operator(\n        lambda id: operators.OutputOperator(datasink, id),\n        lambda operator: operator(table),\n        special=special,\n    )\n","sourceCodeStart":38,"sourceCodeEnd":73,"githubUrl":"https://github.com/pathwaycom/pathway/blob/fa2f74a4649b7c5908690cf60137263d8d80de5f/python/pathway/internals/table_io.py#L38-L73","documentation":"Raised in table_io.table_from_datasource when the datasource's schema declares a column whose source_component is not in supported_components (default: only the payload component). Source components distinguish connector-generated fields (payload) from metadata fields (e.g. system columns like _partition or _offset); only payload columns may flow into a standard table input.","triggerScenarios":"Building a table from a datasource whose schema includes metadata columns (commit timestamps, offsets, partition ids) while calling the generic table_from_datasource without extending supported_components.","commonSituations":"Using Kafka-like connectors that expose system columns; writing a custom DataSource and marking columns with a non-payload source_component; internal API changes where a field's source component was reclassified.","solutions":["Remove the metadata fields from the datasource's schema, or mark them with the payload source component if they are user data","If you intentionally need those fields, pass the appropriate supported_components to table_from_datasource (connector-specific API)","For custom datasources, ensure column definitions use PAYLOAD_SOURCE_COMPONENT for user-visible columns","Check datasource.schema.columns() and each column's source_component to find the offending field"],"exampleFix":"# before (custom datasource exposes a system column)\nclass MySchema(pw.Schema):\n    value: str\n    _offset: int  # declared as metadata component -> ValueError\n\n# after: drop the metadata column or declare it as payload\nclass MySchema(pw.Schema):\n    value: str","handlingStrategy":"validation","validationCode":"from pathway.internals.table_io import PAYLOAD_SOURCE_COMPONENT\n\ndef schema_components_ok(datasource) -> bool:\n    return all(c.source_component in (PAYLOAD_SOURCE_COMPONENT,)\n               for c in datasource.schema.columns().values())","typeGuard":null,"tryCatchPattern":"try:\n    t = table_from_datasource(ds)\nexcept ValueError as e:\n    if 'unsupported source component' in str(e):\n        bad = [n for n, c in ds.schema.columns().items()\n               if c.source_component != PAYLOAD_SOURCE_COMPONENT]\n        raise ValueError(f'metadata columns not allowed: {bad}') from e","preventionTips":["Keep metadata fields out of payload schemas in custom datasources","Pass the connector's supported components explicitly when system columns are intended","Unit-test datasource schemas against PAYLOAD_SOURCE_COMPONENT"],"tags":["pathway","datasource","schema","source-component"],"backgroundTag":null,"analyzedSha":"fa2f74a4649b7c5908690cf60137263d8d80de5f","analyzedAt":"2026-08-15T01:48:17.006Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}