pola-rs/polars · error · NotImplementedError

sink to Iceberg table with '{transform}' partition transform

Error message

sink to Iceberg table with '{transform}' partition transform on '{source_type}'

What it means

The Iceberg sink supports only a subset of partition transforms per source type (identity, year/month/day/hour for temporals, bucket, truncate with known width, etc.). This error means the sink encountered a partition transform it cannot express as a polars expression for the given source column type, and raises NotImplementedError listing the transform and source type.

Source

Thrown at py-polars/src/polars/io/iceberg/_sink.py:327

            elif isinstance(transform, MonthTransform):
                expr = (expr.dt.year() - 1970) * 12 + expr.dt.month() - 1
            elif isinstance(transform, DayTransform):
                expr = expr.cast(pl.Date).cast(pl.Int32)
            else:
                expr = expr.dt.epoch("us") // 3_600_000_000
        elif isinstance(transform, TruncateTransform):
            if isinstance(source_type, (IntegerType, LongType)):
                expr = expr - expr % transform.width
            elif isinstance(source_type, StringType):
                expr = expr.str.slice(0, transform.width)
            elif isinstance(source_type, BinaryType):
                expr = expr.bin.slice(0, transform.width)
            else:
                msg = (
                    "sink to Iceberg table with "
                    f"'{transform}' partition transform on '{source_type}'"
                )
                raise NotImplementedError(msg)
        else:
            msg = f"sink to Iceberg table with '{transform}' partition transform"
            raise NotImplementedError(msg)

        key_name = f"__POLARS_ICEBERG_PARTITION_{field.field_id}"
        while key_name in reserved_names:
            key_name += "_"
        reserved_names.add(key_name)
        exprs.append(expr.alias(key_name))

    return exprs


@dataclass(kw_only=True)
class IcebergSinkState:
    py_catalog_class_module: str
    py_catalog_class_qualname: str

View on GitHub (pinned to fc24390824)

Solutions

Recreate/alter the table's partition spec to use a supported transform for that column type (identity, bucket, truncate, year/month/day/hour as appropriate)
Pick a supported source column type — e.g. use a date/timestamp column for temporal transforms or a string/binary for truncate
Write the transformed value yourself as a regular column and partition with identity transform on it
Check pyiceberg's `field.transform.transform(source_type)` output to see what transform is actually resolved before sinking

Example fix

// before: unsupported transform for the column type
# partition: void('notes') or truncate on a struct column
// after: supported identity transform on an appropriate column
# partition: identity('notes')
lf.sink_iceberg(table)

Defensive patterns

Strategy: validation

Validate before calling

from pyiceberg.transforms import IdentityTransform, BucketTransform, TruncateTransform, YearTransform, MonthTransform, DayTransform, HourTransform
for field in table.spec().fields:
    src = table.metadata.current_schema.find_field(field.source_id)
    t = field.transform
    supported = (IdentityTransform, BucketTransform, TruncateTransform, YearTransform, MonthTransform, DayTransform, HourTransform)
    if not isinstance(t, supported) or t.transform(src.field_type) is None:
        raise ValueError(f"unsupported partition transform {t} on {src.field_type}")

Type guard

def transform_supported(field, schema) -> bool:
    src = schema.find_field(field.source_id)
    try:
        return field.transform.transform(src.field_type) is not None
    except Exception:
        return False

Try / catch

try:
    lf.sink_iceberg(table)
except NotImplementedError as e:
    if 'partition transform' in str(e):
        raise RuntimeError("recreate the table's partition spec using supported transforms") from e
    raise

Prevention

Before sinking, print the spec: [(f.name, str(f.transform)) for f in table.spec().fields]
Avoid void and exotic transforms when a table will be written by polars
Materialize unsupported transforms as your own identity-partitioned columns

When it happens

Trigger: `sink_iceberg` on a table whose partition spec applies a transform unsupported for that source type — e.g. `void` transform, a truncate transform whose width cannot be applied to the column's type, or exotic temporal transforms (e.g. year/day on non-date-time types) — inside `_partition_key_exprs`.

Common situations: Tables partitioned with rare transforms (`void`) created by other engines (Spark/Flink); type evolution changing the source column so an existing transform no longer applies; sinking into a table whose spec was authored for a different schema.

Related errors

AI-assisted analysis of pola-rs/polars@fc24390824 (2026-09-02). Data as JSON: /api/errors/f656332483c7c6fb. Report an issue: GitHub.