pola-rs/polars · error · ValueError

Iceberg sort order is no longer available

Error message

Iceberg sort order {self.sort_order_id} is no longer available

What it means

When finalizing an Iceberg sink write, polars opens a table transaction and checks that the sort order ID recorded earlier still resolves in the table's current metadata. If the sort order was removed (e.g. the table's sort order was replaced/dropped by another writer between scan and sink commit), the write is aborted with this ValueError rather than silently committing data under a dangling sort-order reference.

Solutions

Re-scan the table to pick up fresh metadata so the sink records a currently valid sort_order_id
Check the table's current sort orders (e.g. via pyiceberg `table.metadata.sort_orders`) and re-create/reassign a sort order if it was dropped
Retry the sink after the competing writer finishes, avoiding concurrent metadata-changing operations
If no ordering is required, configure the sink without a sort order so no stale ID is written

Example fix

# before
sink = sink_iceberg(table, sort_order_id=stale_sort_order_id)  # raises at commit
# after
table = catalog.load_table(identifier)  # reload fresh metadata
sort_order_id = table.metadata.default_sort_order_id  # or None
sink = sink_iceberg(table, sort_order_id=sort_order_id)

Defensive patterns

Strategy: try-catch

Validate before calling

so_id = getattr(table.metadata, 'default_sort_order_id', None)
if so_id is not None and table.metadata.sort_order_by_id(so_id) is None:
    so_id = None  # stale sort order; drop it before sinking

Type guard

def sort_order_is_available(table, sort_order_id):
    return sort_order_id is None or table.metadata.sort_order_by_id(sort_order_id) is not None

Try / catch

try:
    lf.sink_iceberg(target)
except ValueError as e:
    if 'no longer available' in str(e):
        table = catalog.load_table(identifier)  # refresh metadata
        lf.sink_iceberg(target)

Prevention

Reload table metadata immediately before sinking rather than reusing a stale handle
Avoid long gaps between scan and sink commits when other writers may alter metadata
Verify the sort order exists in current metadata before configuring the sink
Coordinate concurrent writers to the same Iceberg table

When it happens

Trigger: Calling sink_iceberg/write to an Iceberg table where the sink captured a sort_order_id from table metadata at scan time, but by transaction commit time `table_metadata.sort_order_by_id(self.sort_order_id)` returns None — i.e. the sort order with that ID no longer exists in the table metadata.

Common situations: Concurrent writers rewrote table metadata (dropped or replaced the sort order) while a long-running sink job was in flight; restoring a table from an older metadata/snapshot that lacks the sort order ID; manual metadata surgery or catalog rollback.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of pola-rs/polars@fe841f959e (2026-09-18). Data as JSON: /api/errors/09133adbc3e6d9c9. Report an issue: GitHub.

Appendix: source

Thrown at py-polars/src/polars/io/iceberg/_sink.py:755

        if sys.platform == "win32":
            sinked_files = [
                (
                    f"file://{path[8:]}" if path.startswith("file:///") else path,
                    num_rows,
                    num_bytes,
                    parquet_metadata,
                )
                for path, num_rows, num_bytes, parquet_metadata in sinked_files
            ]

        with table.transaction() as tx:
            if (
                self.sort_order_id is not None
                and tx.table_metadata.sort_order_by_id(self.sort_order_id) is None
            ):
                msg = f"Iceberg sort order {self.sort_order_id} is no longer available"
                raise ValueError(msg)
            self._update_schema(tx)

            if self.mode == "overwrite":
                from pyiceberg.expressions import AlwaysTrue

                tx.delete(AlwaysTrue(), snapshot_properties=self.snapshot_properties)

            if verbose:
                eprint("IcebergSinkState[commit]: begin add_files")

            start_instant = perf_counter()

            _add_files(
                tx,
                sinked_files,
                self.snapshot_properties,
                self.sort_order_id,
            )

View on GitHub (pinned to fe841f959e)