pola-rs/polars · error · TypeError

partition source field {source_id} has non-struct parent

Error message

partition source field {source_id} has non-struct parent

What it means

Polars' Iceberg sink resolves the source expression for a partition field by walking nested accessors down from the top-level table column. This error means an intermediate level of that path is not an Iceberg StructType, so the sink cannot descend into it with `.struct.field(...)` to build the partition column expression. It is a schema-shape limitation of nested partitioning support, not a data error.

Source

Thrown at py-polars/src/polars/io/iceberg/_sink.py:72

        if schema.accessor_for_field(field.source_id).inner is not None
    }


def _partition_source_expr(schema: Schema, source_id: int) -> pl.Expr:
    from pyiceberg.types import StructType

    import polars as pl

    accessor = schema.accessor_for_field(source_id)
    source_field = schema.fields[accessor.position]
    expr = pl.col(source_field.name)

    while accessor.inner is not None:
        accessor = accessor.inner
        source_type = source_field.field_type
        if not isinstance(source_type, StructType):
            msg = f"partition source field {source_id} has non-struct parent"
            raise TypeError(msg)
        source_field = source_type.fields[accessor.position]
        expr = expr.struct.field(source_field.name)

    return expr


def _infer_partition_from_statistics(
    statistics: DataFileStatistics, spec: PartitionSpec, schema: Schema
) -> Record:
    from pyiceberg.partitioning import partition_record_value
    from pyiceberg.typedef import Record

    partition_values: list[Any] = []
    for field in spec.fields:
        aggregate = statistics.column_aggregates.get(field.source_id)
        if aggregate is None:
            partition_values.append(None)
            continue

View on GitHub (pinned to fc24390824)

Solutions

1. Recreate or alter the table's partition spec so partition fields sit at the top level or inside plain struct columns only
2. Flatten the struct column in the DataFrame (`.unnest(...)` or select the nested field) and sink to a table partitioned on the flattened top-level column
3. Check `table.spec().fields` and `schema.accessor_for_field(source_id)` with pyiceberg to confirm what the partition path actually traverses before sinking
4. Upgrade polars; nested partition support may have expanded in newer versions

Example fix

// before: partition spec on a field nested inside a list column
# partition field source_id -> ListType element
// after: repartition the table on a top-level column
# iceberg partition spec: identity transform on top-level column 'category'
lf.sink_iceberg(table)

from pyiceberg.types import StructType
schema = table.metadata.current_schema
for field in table.spec().fields:
    accessor = schema.accessor_for_field(field.source_id)
    node, ft = accessor, schema.fields[accessor.position]
    while node.inner is not None:
        node = node.inner
        if not isinstance(ft.field_type, StructType):
            raise ValueError(f"partition field {field.source_id} has non-struct parent")
        ft = ft.field_type.fields[node.position]

def is_struct_parent_path(schema, source_id) -> bool:
    from pyiceberg.types import StructType
    accessor = schema.accessor_for_field(source_id)
    f = schema.fields[accessor.position]
    while accessor.inner is not None:
        accessor = accessor.inner
        if not isinstance(f.field_type, StructType):
            return False
        f = f.field_type.fields[accessor.position]
    return True

When it happens

Trigger: Sinking a LazyFrame to an Iceberg table whose partition spec references a source_id whose parent accessor chain passes through a non-struct type (e.g. a partition defined on a field nested inside a list or map rather than a struct). Raised inside `_partition_source_expr` while iterating `accessor.inner`.

Common situations: Tables partitioned on fields nested in list/map columns (Iceberg allows this, polars does not); metadata drift where the table's partition spec targets a nested column of unexpected type; hand-edited or migrated Iceberg metadata where source_id/position mapping no longer lines up with a struct.

Related errors

• Cannot infer partition value from Parquet metadata for parti
• sink to Iceberg table with partition field '{field.name}' on
• sink to Iceberg table with '{transform}' partition transform
• sink to Iceberg table with '{transform}' partition transform
• schema_mode='overwrite' is not supported for partitioned Ice

AI-assisted analysis of pola-rs/polars@fc24390824 (2026-09-02). Data as JSON: /api/errors/fc604705e8a30aa7. Report an issue: GitHub.