pola-rs/polars · error · NotImplementedError

schema_mode='overwrite' is not supported for partitioned Ice

Error message

schema_mode='overwrite' is not supported for partitioned Iceberg tables

What it means

`schema_mode='overwrite'` in `sink_iceberg` replaces the table's schema, which conflicts with an existing partition spec derived from the old schema's fields. Because partition fields depend on source column ids/types, polars refuses this combination with NotImplementedError rather than producing an inconsistent table.

Source

Thrown at py-polars/src/polars/io/iceberg/_sink.py:499

        evolved_schema = transaction.table_metadata.schema()
        source_schema = pyarrow_to_schema(
            self._get_source_schema(), name_mapping=evolved_schema.name_mapping
        )
        return schema_to_pyarrow(source_schema)

    def _attach_resolved_sink(self, plf: PyLazyFrame) -> PyLazyFrame:
        from pyiceberg.table import TableProperties
        from pyiceberg.utils.properties import property_as_bool, property_as_int

        import polars as pl

        table = self.table()
        table_metadata = table.metadata
        table_properties = table_metadata.properties

        if self.schema_mode == "overwrite" and table.spec().fields:
            msg = "schema_mode='overwrite' is not supported for partitioned Iceberg tables"
            raise NotImplementedError(msg)

        partition_key_exprs = _partition_key_exprs(table, self.source_schema)

        if table.sort_order().fields:
            msg = "sink to Iceberg table with sort order"
            raise NotImplementedError(msg)

        if location_provider_impl := table_properties.get(
            TableProperties.WRITE_PY_LOCATION_PROVIDER_IMPL
        ):
            msg = (
                "sink to Iceberg table with custom location provider"
                f" '{location_provider_impl}'"
            )
            raise NotImplementedError(msg)

        object_storage_enabled = property_as_bool(
            table_properties,

View on GitHub (pinned to fc24390824)

Solutions

Use `schema_mode='append'` (the default) if the data conforms to the existing schema
Drop or recreate the table: create a new table with the desired schema and partition spec, then sink into it
If you truly need a new schema, create a fresh table with matching partition spec fields that reference the new schema, then sink with overwrite on the new (unpartitioned-target) workflow
Keep schema evolution within compatible types so append mode works instead of overwrite

Example fix

// before: overwrite against a partitioned table
lf.sink_iceberg(table, schema_mode='overwrite')
// after: append, or recreate the table with the new schema/spec first
lf.sink_iceberg(table, schema_mode='append')  # or create new table, then sink

Defensive patterns

Strategy: validation

Validate before calling

if table.spec().fields and schema_mode == 'overwrite':
    raise ValueError("schema_mode='overwrite' cannot be used on a partitioned Iceberg table")

Try / catch

try:
    lf.sink_iceberg(table, schema_mode=schema_mode)
except NotImplementedError as e:
    if "schema_mode='overwrite'" in str(e):
        raise RuntimeError("use append mode, or recreate the table with the desired schema") from e
    raise

Prevention

Default to schema_mode='append' when sinking into existing partitioned tables
If schema changes are needed, create a new table with the new schema and partition spec instead of overwriting
Check `len(table.spec().fields)` before choosing overwrite

When it happens

Trigger: Calling `lf.sink_iceberg(table, schema_mode='overwrite')` where the target Iceberg table has a non-empty partition spec (`table.spec().fields` is non-empty).

Common situations: Evolving a partitioned table's schema by overwrite instead of append; copying a sink configuration written for an unpartitioned table and reusing it against a partitioned one; migrating tables where the spec wasn't dropped first.

Related errors

AI-assisted analysis of pola-rs/polars@fc24390824 (2026-09-02). Data as JSON: /api/errors/eb7ff3d763b39c5b. Report an issue: GitHub.