{"record":{"id":"68656507e37536e1","repo":"pola-rs/polars","slug":"dataframe-contains-unsupported-data-types-overla","errorCode":null,"errorMessage":"dataframe contains unsupported data types: {overlap!r}","messagePattern":"dataframe contains unsupported data types: (.+?)","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"py-polars/src/polars/io/delta/_utils.py","lineNumber":107,"sourceCode":"    return dl_tbl\n\n\ndef _check_if_delta_available() -> None:\n    if not _DELTALAKE_AVAILABLE:\n        msg = \"deltalake is not installed\\n\\nPlease run: pip install deltalake\"\n        raise ModuleNotFoundError(msg)\n\n\ndef _check_for_unsupported_types(dtypes: list[DataType]) -> None:\n    schema_dtypes = unpack_dtypes(*dtypes)\n    unsupported_types = {Time, Null}\n    # Note that this overlap check does NOT work correctly for Categorical, so\n    # if Categorical is added back to unsupported_types a different check will\n    # need to be used.\n\n    if overlap := schema_dtypes & unsupported_types:\n        msg = f\"dataframe contains unsupported data types: {overlap!r}\"\n        raise TypeError(msg)\n\n\ndef _extract_table_statistics_from_delta_add_actions(\n    add_actions_df: DataFrame,\n    *,\n    filter_columns: list[str],\n    schema: SchemaDict,\n    verbose: bool,\n) -> DataFrame | None:\n    import polars as pl\n\n    if \"num_records\" not in add_actions_df:\n        if verbose:\n            eprint(\n                \"scan_delta: statistics load failed: 'num_records' column not present\"\n            )\n\n        return None","sourceCodeStart":89,"sourceCodeEnd":125,"githubUrl":"https://github.com/pola-rs/polars/blob/df599052daf96e7a9cc30a3b0c6bd25d6947e3c0/py-polars/src/polars/io/delta/_utils.py#L89-L125","documentation":"Raised by _check_for_unsupported_types when writing a DataFrame whose schema (including nested fields, via unpack_dtypes) contains Time or Null dtypes, which the Delta format cannot represent. The check runs before any I/O so the write fails fast with the offending set of types.","triggerScenarios":"pl.write_delta() with a frame containing a pl.Time column or a column typed pl.Null (e.g. created by pl.lit(None) without a dtype or select(pl.lit(None))); nested Time/Null inside structs/lists is also caught by unpack_dtypes.","commonSituations":"ETL jobs adding an all-NULL marker column; time-of-day data ingested from CSV/JDBC as pl.Time; schema-on-write validation catching leftover placeholder columns.","solutions":["Cast Time columns to a supported representation: pl.Datetime (with a reference date), pl.String ('%H:%M:%S'), or pl.Int64 (microseconds since midnight)","Drop all-Null columns (df.drop(...)) or give them a concrete dtype, e.g. pl.lit(None, dtype=pl.String)","Re-run after fixing each dtype named in the error set"],"exampleFix":"# before\ndf = df.with_columns(pl.lit(None).alias(\"note\"), pl.col(\"shift_start\").cast(pl.Time))\ndf.write_delta(\"./tbl\")\n\n# after\ndf = df.with_columns(\n    pl.lit(None, dtype=pl.String).alias(\"note\"),\n    shift_start_us=pl.col(\"shift_start\").cast(pl.Time).dt.total_microseconds(),\n).drop(\"shift_start\")\ndf.write_delta(\"./tbl\")","handlingStrategy":"validation","validationCode":"import polars as pl\nfrom polars.datatypes import unpack_dtypes\n\nbad = unpack_dtypes(*df.schema.dtypes()) & {pl.Time, pl.Null}\nif bad:\n    raise TypeError(f\"fix these dtypes before write_delta: {bad}\")","typeGuard":"def is_delta_writable(df: pl.DataFrame) -> bool:\n    from polars.datatypes import unpack_dtypes\n    return not (unpack_dtypes(*df.schema.dtypes()) & {pl.Time, pl.Null})","tryCatchPattern":null,"preventionTips":["Cast pl.Time to Int64 microseconds or String before write_delta","Type all-NULL literals: pl.lit(None, dtype=pl.String) instead of bare pl.lit(None)","Validate schemas in a pre-write lint step for ETL pipelines"],"tags":["delta-lake","dtype","write","type-error"],"backgroundTag":null,"analyzedSha":"df599052daf96e7a9cc30a3b0c6bd25d6947e3c0","analyzedAt":"2026-08-16T12:10:03.978Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}