pola-rs/polars · error · DuplicateError

duplicate column name: {column_info.name}

Error message

duplicate column name: {column_info.name}

What it means

`TableInfo.get_polars_schema()` converts Unity Catalog column metadata into a `pl.Schema`, which is a dict keyed by column name — names must be unique. If the catalog's `columns` list contains the same name twice, the second assignment collides and polars raises `DuplicateError`. This indicates ambiguous/corrupted table metadata on the catalog side, not a polars bug.

Source

Thrown at py-polars/src/polars/catalog/unity/models.py:85

        """
        Get the native polars schema of this table.

        .. warning::
            This functionality is considered **unstable**. It may be changed
            at any point without it being considered a breaking change.
        """
        issue_unstable_warning(
            "`get_polars_schema` functionality is considered unstable."
        )
        if self.columns is None:
            return None

        schema = Schema()

        for column_info in self.columns:
            if column_info.name in schema:
                msg = f"duplicate column name: {column_info.name}"
                raise DuplicateError(msg)
            schema[column_info.name] = column_info.get_polars_dtype()

        return schema


@dataclass
class ColumnInfo:
    """Information for a column within a catalog table."""

    name: str
    type_name: str
    type_text: str
    type_json: str
    position: int | None
    comment: str | None
    partition_index: int | None

    def get_polars_dtype(self) -> DataType:

View on GitHub (pinned to 9b5d73fd00)

Solutions

1. Find the duplicate: `names = [c.name for c in table_info.columns]; print([n for n in set(names) if names.count(n) > 1])`.
2. Fix the table in the catalog: `ALTER TABLE ... RENAME COLUMN` the duplicate, or re-register the table with unique column names.
3. If the physical files are fine, read them directly with `pl.scan_delta(path, schema=...)` supplying an explicit deduplicated schema.

Example fix

# before
schema = table_info.get_polars_schema()  # DuplicateError: duplicate column name: 'id'

# after: locate and fix the duplicate in the catalog, or read files with explicit schema
schema = pl.Schema({"id": pl.Int64, "id_1": pl.Int64, "value": pl.Float64})
lf = pl.scan_delta("s3://bucket/path/to/table", schema=schema)

from collections import Counter

names = [c.name for c in table_info.columns or []]
dupes = [n for n, count in Counter(names).items() if count > 1]
if dupes:
    raise ValueError(f"duplicate columns in catalog metadata: {dupes}")
schema = table_info.get_polars_schema()

from polars.exceptions import DuplicateError

try:
    schema = table_info.get_polars_schema()
except DuplicateError as err:
    names = [c.name for c in table_info.columns]
    # proceed with files + explicit deduplicated schema
    schema = pl.Schema({f"{n}_{i}" if names[:i].count(n) else n: pl.String for i, n in enumerate(names)})

When it happens

Trigger: Calling `table_info.get_polars_schema()` directly, or `pl.scan_unity_catalog`/`read_unity_catalog` deriving the schema, on a table whose ColumnInfo list has a repeated `name` (including case-collisions the catalog considers distinct).

Common situations: Schema-evolution edge cases or manual metadata registration that produced two identical column names; non-Delta foreign tables imported into UC with colliding names; case-sensitive duplicates ('ID' vs 'id') that the source system allowed.

Related errors

• Deserialization from JSON not implemented for {adt:?}
• The external API has a non-utf8 as format
• unexpected dtype when deserializing ndjson
• mapping item must be a datatype or datatype expression; foun
• cannot {operation}: no storage_location found

AI-assisted analysis of pola-rs/polars@9b5d73fd00 (2026-08-19). Data as JSON: /api/errors/de3c28bf4e770858. Report an issue: GitHub.