Pandas dataframe contains non-unique indices and/or column n

Error message

Pandas dataframe contains non-unique indices and/or column names. Polars dataframes require unique string names for columns.

What it means

Before converting a pandas DataFrame, polars stringifies column names (and, with include_index, index-level names) and requires them to be unique — Polars identifies columns by unique string names. If stringification collapses entries (duplicate labels like two 'a' columns, or mixed types that stringify equal, e.g. int 1 and str '1'), this ValueError is raised instead of producing an invalid schema.

Source

Thrown at py-polars/src/polars/_utils/construction/dataframe.py:1077


def _check_pandas_columns(data: pd.DataFrame, *, include_index: bool) -> None:
    """Check pandas dataframe columns can be converted to polars."""
    stringified_cols: set[str] = {str(col) for col in data.columns}
    stringified_index: set[str] = (
        {str(idx) for idx in data.index.names} if include_index else set()
    )

    non_unique_cols: bool = len(stringified_cols) < len(data.columns)
    non_unique_indices: bool = (
        (len(stringified_index) < len(data.index.names)) if include_index else False
    )
    if non_unique_cols or non_unique_indices:
        msg = (
            "Pandas dataframe contains non-unique indices and/or column names. "
            "Polars dataframes require unique string names for columns."
        )
        raise ValueError(msg)

    overlapping_cols_and_indices: set[str] = stringified_cols & stringified_index
    if len(overlapping_cols_and_indices) > 0:
        msg = "Pandas indices and column names must not overlap."
        raise ValueError(msg)


def pandas_to_pydf(
    data: pd.DataFrame,
    schema: SchemaDefinition | None = None,
    *,
    schema_overrides: SchemaDict | None = None,
    strict: bool = True,
    rechunk: bool = True,
    nan_to_null: bool = True,
    include_index: bool = False,
) -> PyDataFrame:
    """Construct a PyDataFrame from a pandas DataFrame."""

View on GitHub (pinned to df599052da)

Solutions

Deduplicate before converting: `df = df.loc[:, ~df.columns.duplicated()]` (keeps first occurrence)
Rename to unique names: `df.columns = [f"{c}_{i}" if duplicated else c ...]` or `df.add_suffix("_2")` where appropriate
Fix the upstream pandas operation (named aggregation, join with lsuffix/rsuffix) so duplicates never appear
For MultiIndex, flatten first: `df.columns = ["_".join(map(str, lvl)) for lvl in df.columns]`

Example fix

# before
import pandas as pd, polars as pl
pdf = pd.DataFrame([[1, 2]], columns=["a", "a"])
pl.from_pandas(pdf)  # ValueError: Pandas dataframe contains non-unique indices and/or column names ...

# after
pdf = pdf.loc[:, ~pdf.columns.duplicated()]
# or make unique explicitly:
pdf.columns = ["a", "a_2"]
pl.from_pandas(pdf)

Defensive patterns

Strategy: validation

Validate before calling

import pandas as pd

assert df.columns.is_unique, f"duplicate pandas columns: {df.columns[df.columns.duplicated()].tolist()}"
assert df.index.names == [None] or len(set(map(str, df.index.names))) == len(df.index.names)
pl.from_pandas(df)

Type guard

import pandas as pd

def pandas_cols_unique(df: pd.DataFrame) -> bool:
    """True if stringified column names (and index names) are unique."""
    cols_ok = len({str(c) for c in df.columns}) == len(df.columns)
    idx_ok = len({str(n) for n in df.index.names}) == len(df.index.names)
    return cols_ok and idx_ok

Try / catch

try:
    pl.from_pandas(df)
except ValueError as e:
    if "non-unique" not in str(e):
        raise
    df = df.loc[:, ~df.columns.duplicated()]
    pl.from_pandas(df)

Prevention

Assert df.columns.is_unique before conversion in ingestion helpers
Use lsuffix/rsuffix on pandas joins and named aggregations so duplicates never occur
Flatten MultiIndex columns to underscore-joined strings before converting

When it happens

Trigger: `pl.from_pandas(df)` where df has duplicate column labels (common after joins/aggregations without suffixes or read_csv with repeated headers); or include_index conversion where two index levels stringify to the same name; columns whose names are distinct in pandas but equal after str().

Common situations: pandas joins without lsuffix/rsuffix producing duplicated names; groupby/agg that keeps the grouping key plus a same-named aggregate; MultiIndex flattened to repeated level names; numeric column labels 0..n alongside string '0'.

Related errors

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/eee877f216c02cbd. Report an issue: GitHub.