{"record":{"id":"f656332483c7c6fb","repo":"pola-rs/polars","slug":"sink-to-iceberg-table-with-transform-partition","errorCode":null,"errorMessage":"sink to Iceberg table with '{transform}' partition transform on '{source_type}'","messagePattern":"sink to Iceberg table with '(.+?)' partition transform on '(.+?)'","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"py-polars/src/polars/io/iceberg/_sink.py","lineNumber":327,"sourceCode":"            elif isinstance(transform, MonthTransform):\n                expr = (expr.dt.year() - 1970) * 12 + expr.dt.month() - 1\n            elif isinstance(transform, DayTransform):\n                expr = expr.cast(pl.Date).cast(pl.Int32)\n            else:\n                expr = expr.dt.epoch(\"us\") // 3_600_000_000\n        elif isinstance(transform, TruncateTransform):\n            if isinstance(source_type, (IntegerType, LongType)):\n                expr = expr - expr % transform.width\n            elif isinstance(source_type, StringType):\n                expr = expr.str.slice(0, transform.width)\n            elif isinstance(source_type, BinaryType):\n                expr = expr.bin.slice(0, transform.width)\n            else:\n                msg = (\n                    \"sink to Iceberg table with \"\n                    f\"'{transform}' partition transform on '{source_type}'\"\n                )\n                raise NotImplementedError(msg)\n        else:\n            msg = f\"sink to Iceberg table with '{transform}' partition transform\"\n            raise NotImplementedError(msg)\n\n        key_name = f\"__POLARS_ICEBERG_PARTITION_{field.field_id}\"\n        while key_name in reserved_names:\n            key_name += \"_\"\n        reserved_names.add(key_name)\n        exprs.append(expr.alias(key_name))\n\n    return exprs\n\n\n@dataclass(kw_only=True)\nclass IcebergSinkState:\n    py_catalog_class_module: str\n    py_catalog_class_qualname: str\n","sourceCodeStart":309,"sourceCodeEnd":345,"githubUrl":"https://github.com/pola-rs/polars/blob/fc24390824009e8fc5b7870e256273c1b140a019/py-polars/src/polars/io/iceberg/_sink.py#L309-L345","documentation":"The Iceberg sink supports only a subset of partition transforms per source type (identity, year/month/day/hour for temporals, bucket, truncate with known width, etc.). This error means the sink encountered a partition transform it cannot express as a polars expression for the given source column type, and raises NotImplementedError listing the transform and source type.","triggerScenarios":"`sink_iceberg` on a table whose partition spec applies a transform unsupported for that source type — e.g. `void` transform, a truncate transform whose width cannot be applied to the column's type, or exotic temporal transforms (e.g. year/day on non-date-time types) — inside `_partition_key_exprs`.","commonSituations":"Tables partitioned with rare transforms (`void`) created by other engines (Spark/Flink); type evolution changing the source column so an existing transform no longer applies; sinking into a table whose spec was authored for a different schema.","solutions":["Recreate/alter the table's partition spec to use a supported transform for that column type (identity, bucket, truncate, year/month/day/hour as appropriate)","Pick a supported source column type — e.g. use a date/timestamp column for temporal transforms or a string/binary for truncate","Write the transformed value yourself as a regular column and partition with identity transform on it","Check pyiceberg's `field.transform.transform(source_type)` output to see what transform is actually resolved before sinking"],"exampleFix":"// before: unsupported transform for the column type\n# partition: void('notes') or truncate on a struct column\n// after: supported identity transform on an appropriate column\n# partition: identity('notes')\nlf.sink_iceberg(table)","handlingStrategy":"validation","validationCode":"from pyiceberg.transforms import IdentityTransform, BucketTransform, TruncateTransform, YearTransform, MonthTransform, DayTransform, HourTransform\nfor field in table.spec().fields:\n    src = table.metadata.current_schema.find_field(field.source_id)\n    t = field.transform\n    supported = (IdentityTransform, BucketTransform, TruncateTransform, YearTransform, MonthTransform, DayTransform, HourTransform)\n    if not isinstance(t, supported) or t.transform(src.field_type) is None:\n        raise ValueError(f\"unsupported partition transform {t} on {src.field_type}\")","typeGuard":"def transform_supported(field, schema) -> bool:\n    src = schema.find_field(field.source_id)\n    try:\n        return field.transform.transform(src.field_type) is not None\n    except Exception:\n        return False","tryCatchPattern":"try:\n    lf.sink_iceberg(table)\nexcept NotImplementedError as e:\n    if 'partition transform' in str(e):\n        raise RuntimeError(\"recreate the table's partition spec using supported transforms\") from e\n    raise","preventionTips":["Before sinking, print the spec: [(f.name, str(f.transform)) for f in table.spec().fields]","Avoid void and exotic transforms when a table will be written by polars","Materialize unsupported transforms as your own identity-partitioned columns"],"tags":["iceberg","partitioning","transform","unsupported","polars"],"backgroundTag":"unsupported-partition-transform","analyzedSha":"fc24390824009e8fc5b7870e256273c1b140a019","analyzedAt":"2026-09-02T20:08:03.494Z","contentChangedAt":"2026-09-02T20:08:03.494Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}