{"record":{"id":"e36767d5116d23c9","repo":"pola-rs/polars","slug":"schema-mode-overwrite-requires-mode-overwrite","errorCode":null,"errorMessage":"schema_mode='overwrite' requires mode='overwrite'","messagePattern":"schema_mode='overwrite' requires mode='overwrite'","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"py-polars/src/polars/io/iceberg/_sink.py","lineNumber":67,"sourceCode":"    sink_uuid_str: str\n\n    table_: NoPickleOption[pyiceberg.table.Table]\n    source_schema: pa.Schema | None\n    commit_result_df: NoPickleOption[pl.DataFrame]\n\n    @staticmethod\n    def new(\n        target: str | pyiceberg.table.Table,\n        *,\n        mode: Literal[\"append\", \"overwrite\"] = \"append\",\n        schema_mode: Literal[\"merge\", \"overwrite\"] | None = None,\n        snapshot_properties: dict[str, str] | None = None,\n        catalog: pyiceberg.catalog.Catalog | IcebergCatalogConfig | None = None,\n        storage_options: StorageOptionsDict | None = None,\n    ) -> IcebergSinkState:\n        if schema_mode == \"overwrite\" and mode != \"overwrite\":\n            msg = \"schema_mode='overwrite' requires mode='overwrite'\"\n            raise ValueError(msg)\n\n        catalog_config = (\n            (\n                IcebergCatalogConfig._from_api_parameter_or_environment_default(\n                    catalog,\n                    fn_name=\"sink_iceberg\",\n                )\n            )\n            if isinstance(target, str)\n            else (\n                IcebergCatalogConfig(\n                    class_=type(target.catalog),\n                    name=target.catalog.name,\n                    properties=target.catalog.properties,\n                )\n            )\n        )\n","sourceCodeStart":49,"sourceCodeEnd":85,"githubUrl":"https://github.com/pola-rs/polars/blob/4db92c12c08c390f95b517a00ae0afe818e1abf6/py-polars/src/polars/io/iceberg/_sink.py#L49-L85","documentation":"Raised by polars' Iceberg sink when sink_iceberg is configured with schema_mode='overwrite' while mode is left at its default 'append' (the guard runs first thing in IcebergSinkState.new, _sink.py:65-67). Overwriting a table's schema is only coherent when the write also replaces the table contents — the same coupling pyiceberg and Spark enforce between data mode and schema mode — so append + schema-overwrite is rejected as contradictory. The check is eager: it fires while the sink state is built, before any data is read or written.","triggerScenarios":"df.sink_iceberg(target, schema_mode='overwrite') with mode omitted (defaults to 'append'); any call pairing schema_mode='overwrite' with mode='append'. Valid pairings: mode='overwrite' accepts schema_mode 'merge'/'overwrite'/None; mode='append' accepts only 'merge'/None.","commonSituations":"Porting Spark writeTo(...).option('mergeSchema'/'overwriteSchema', ...) or pyiceberg writer configs where schema mode is set independently of save mode; pipeline templates that always set schema_mode='overwrite' to force schema sync; assuming schema_mode is pure DDL that leaves append behavior untouched.","solutions":["Add mode='overwrite' alongside schema_mode='overwrite' when you intend to replace both table contents and schema.","If the goal is appending rows while evolving the schema, use schema_mode='merge' with the default mode='append'.","If no schema handling is needed, omit schema_mode entirely."],"exampleFix":"# before\nlf.sink_iceberg(\"catalog.db.events\", schema_mode=\"overwrite\")\n\n# after\nlf.sink_iceberg(\"catalog.db.events\", mode=\"overwrite\", schema_mode=\"overwrite\")","handlingStrategy":"validation","validationCode":"from typing import Literal\n\nMode = Literal[\"append\", \"overwrite\"]\nSchemaMode = Literal[\"merge\", \"overwrite\"] | None\n\ndef validate_sink_iceberg_options(mode: Mode, schema_mode: SchemaMode) -> None:\n    if schema_mode == \"overwrite\" and mode != \"overwrite\":\n        msg = \"schema_mode='overwrite' requires mode='overwrite'\"\n        raise ValueError(msg)","typeGuard":"from typing import Literal, TypeGuard\n\nMode = Literal[\"append\", \"overwrite\"]\nSchemaMode = Literal[\"merge\", \"overwrite\"] | None\n\ndef sink_options_consistent(mode: Mode, schema_mode: SchemaMode) -> TypeGuard[bool]:\n    return schema_mode != \"overwrite\" or mode == \"overwrite\"","tryCatchPattern":"try:\n    df.sink_iceberg(\"catalog.db.tbl\", schema_mode=\"overwrite\")\nexcept ValueError as exc:\n    if \"schema_mode\" in str(exc):\n        # option pairing bug: fix mode/schema_mode and retry\n        ...\n    raise","preventionTips":["Treat schema_mode='overwrite' as implying mode='overwrite'; always set the pair together.","For append pipelines that evolve schema, reach for schema_mode='merge', not 'overwrite'.","Validate sink options before launching the write — the ValueError is raised up front in IcebergSinkState.new, so a pre-flight check costs nothing.","When porting Spark/pyiceberg writers, re-check mode vs schema-mode coupling; engines differ in what combinations they allow."],"tags":["polars","iceberg","pyiceberg","python","sink","data-lake"],"backgroundTag":"incompatible-parameter-combination","analyzedSha":"4db92c12c08c390f95b517a00ae0afe818e1abf6","analyzedAt":"2026-08-23T02:54:19.138Z","contentChangedAt":"2026-08-23T02:54:19.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}