pathwaycom/pathway · error · ValueError

The field '{column_name}' in the schema of the source '{data

Error message

The field '{column_name}' in the schema of the source '{datasource.name}' has unsupported source component: '{column_data.source_component}'

What it means

Raised in table_io.table_from_datasource when the datasource's schema declares a column whose source_component is not in supported_components (default: only the payload component). Source components distinguish connector-generated fields (payload) from metadata fields (e.g. system columns like _partition or _offset); only payload columns may flow into a standard table input.

Source

Thrown at python/pathway/internals/table_io.py:56

    *,
    supported_components: Iterable[str] = ...,
) -> TTable: ...


def table_from_datasource(
    datasource: datasources.DataSource,
    debug_datasource: datasources.StaticDataSource | None = None,
    table_cls: type[tables.Table] = tables.Table,
    *,
    supported_components: Iterable[str] = (PAYLOAD_SOURCE_COMPONENT,),
) -> tables.Table:
    for column_name, column_data in datasource.schema.columns().items():
        if column_data.source_component not in supported_components:
            error_message = (
                f"The field '{column_name}' in the schema of the source '{datasource.name}' "
                f"has unsupported source component: '{column_data.source_component}'"
            )
            raise ValueError(error_message)

    return parse_graphs.G.add_operator(
        lambda id: operators.InputOperator(datasource, id, debug_datasource),
        lambda operator: operator(table_cls),
    )


def table_to_datasink(
    table: tables.Table, datasink: datasinks.DataSink, *, special: bool = False
) -> operators.OutputOperator:
    datasink.check_sort_by_columns(table)
    return parse_graphs.G.add_operator(
        lambda id: operators.OutputOperator(datasink, id),
        lambda operator: operator(table),
        special=special,
    )

View on GitHub (pinned to fa2f74a464)

Solutions

1. Remove the metadata fields from the datasource's schema, or mark them with the payload source component if they are user data
2. If you intentionally need those fields, pass the appropriate supported_components to table_from_datasource (connector-specific API)
3. For custom datasources, ensure column definitions use PAYLOAD_SOURCE_COMPONENT for user-visible columns
4. Check datasource.schema.columns() and each column's source_component to find the offending field

Example fix

# before (custom datasource exposes a system column)
class MySchema(pw.Schema):
    value: str
    _offset: int  # declared as metadata component -> ValueError

# after: drop the metadata column or declare it as payload
class MySchema(pw.Schema):
    value: str

from pathway.internals.table_io import PAYLOAD_SOURCE_COMPONENT

def schema_components_ok(datasource) -> bool:
    return all(c.source_component in (PAYLOAD_SOURCE_COMPONENT,)
               for c in datasource.schema.columns().values())

try:
    t = table_from_datasource(ds)
except ValueError as e:
    if 'unsupported source component' in str(e):
        bad = [n for n, c in ds.schema.columns().items()
               if c.source_component != PAYLOAD_SOURCE_COMPONENT]
        raise ValueError(f'metadata columns not allowed: {bad}') from e

When it happens

Trigger: Building a table from a datasource whose schema includes metadata columns (commit timestamps, offsets, partition ids) while calling the generic table_from_datasource without extending supported_components.

Common situations: Using Kafka-like connectors that expose system columns; writing a custom DataSource and marking columns with a non-payload source_component; internal API changes where a field's source component was reclassified.

Related errors

• Failed to detect the region of S3 bucket {bucket!r} (HTTP st
• SchemaRegistryHeader.value must be a str, got {type(self.val
• argument {name} has incorrect schema
• DateTimeNaive cannot contain timezone information. Use pw.Da
• DateTimeUtc must contain timezone information. Use pw.DateTi

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/66efa76c8d145e68. Report an issue: GitHub.