{"record":{"id":"09133adbc3e6d9c9","repo":"pola-rs/polars","slug":"iceberg-sort-order-self-sort-order-id-is-no-longer-available","errorCode":null,"errorMessage":"Iceberg sort order {self.sort_order_id} is no longer available","messagePattern":"Iceberg sort order (.+?) is no longer available","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"py-polars/src/polars/io/iceberg/_sink.py","lineNumber":755,"sourceCode":"\n        if sys.platform == \"win32\":\n            sinked_files = [\n                (\n                    f\"file://{path[8:]}\" if path.startswith(\"file:///\") else path,\n                    num_rows,\n                    num_bytes,\n                    parquet_metadata,\n                )\n                for path, num_rows, num_bytes, parquet_metadata in sinked_files\n            ]\n\n        with table.transaction() as tx:\n            if (\n                self.sort_order_id is not None\n                and tx.table_metadata.sort_order_by_id(self.sort_order_id) is None\n            ):\n                msg = f\"Iceberg sort order {self.sort_order_id} is no longer available\"\n                raise ValueError(msg)\n            self._update_schema(tx)\n\n            if self.mode == \"overwrite\":\n                from pyiceberg.expressions import AlwaysTrue\n\n                tx.delete(AlwaysTrue(), snapshot_properties=self.snapshot_properties)\n\n            if verbose:\n                eprint(\"IcebergSinkState[commit]: begin add_files\")\n\n            start_instant = perf_counter()\n\n            _add_files(\n                tx,\n                sinked_files,\n                self.snapshot_properties,\n                self.sort_order_id,\n            )","sourceCodeStart":737,"sourceCodeEnd":773,"githubUrl":"https://github.com/pola-rs/polars/blob/fe841f959ef4d2ceefc05a310d33ed7b1ab24e5e/py-polars/src/polars/io/iceberg/_sink.py#L737-L773","documentation":"When finalizing an Iceberg sink write, polars opens a table transaction and checks that the sort order ID recorded earlier still resolves in the table's current metadata. If the sort order was removed (e.g. the table's sort order was replaced/dropped by another writer between scan and sink commit), the write is aborted with this ValueError rather than silently committing data under a dangling sort-order reference.","triggerScenarios":"Calling sink_iceberg/write to an Iceberg table where the sink captured a sort_order_id from table metadata at scan time, but by transaction commit time `table_metadata.sort_order_by_id(self.sort_order_id)` returns None — i.e. the sort order with that ID no longer exists in the table metadata.","commonSituations":"Concurrent writers rewrote table metadata (dropped or replaced the sort order) while a long-running sink job was in flight; restoring a table from an older metadata/snapshot that lacks the sort order ID; manual metadata surgery or catalog rollback.","solutions":["Re-scan the table to pick up fresh metadata so the sink records a currently valid sort_order_id","Check the table's current sort orders (e.g. via pyiceberg `table.metadata.sort_orders`) and re-create/reassign a sort order if it was dropped","Retry the sink after the competing writer finishes, avoiding concurrent metadata-changing operations","If no ordering is required, configure the sink without a sort order so no stale ID is written"],"exampleFix":"# before\nsink = sink_iceberg(table, sort_order_id=stale_sort_order_id)  # raises at commit\n# after\ntable = catalog.load_table(identifier)  # reload fresh metadata\nsort_order_id = table.metadata.default_sort_order_id  # or None\nsink = sink_iceberg(table, sort_order_id=sort_order_id)","handlingStrategy":"try-catch","validationCode":"so_id = getattr(table.metadata, 'default_sort_order_id', None)\nif so_id is not None and table.metadata.sort_order_by_id(so_id) is None:\n    so_id = None  # stale sort order; drop it before sinking","typeGuard":"def sort_order_is_available(table, sort_order_id):\n    return sort_order_id is None or table.metadata.sort_order_by_id(sort_order_id) is not None","tryCatchPattern":"try:\n    lf.sink_iceberg(target)\nexcept ValueError as e:\n    if 'no longer available' in str(e):\n        table = catalog.load_table(identifier)  # refresh metadata\n        lf.sink_iceberg(target)","preventionTips":["Reload table metadata immediately before sinking rather than reusing a stale handle","Avoid long gaps between scan and sink commits when other writers may alter metadata","Verify the sort order exists in current metadata before configuring the sink","Coordinate concurrent writers to the same Iceberg table"],"tags":["iceberg","concurrency","metadata","sink"],"backgroundTag":"invalid-state-transition","analyzedSha":"fe841f959ef4d2ceefc05a310d33ed7b1ab24e5e","analyzedAt":"2026-09-18T22:14:11.667Z","contentChangedAt":"2026-09-18T22:14:11.667Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}