pola-rs/polars · error

Pandas indices and column names must not overlap.

Error message

Pandas indices and column names must not overlap.

What it means

When converting a pandas DataFrame with index inclusion (pl.from_pandas(..., include_index=True) or equivalent constructor paths), polars must place both the index levels and the columns into one flat, uniquely-named schema. If any stringified index-level name equals a column name, the sets overlap and this ValueError is raised before conversion starts.

Source

Thrown at py-polars/src/polars/_utils/construction/dataframe.py:1082

    stringified_index: set[str] = (
        {str(idx) for idx in data.index.names} if include_index else set()
    )

    non_unique_cols: bool = len(stringified_cols) < len(data.columns)
    non_unique_indices: bool = (
        (len(stringified_index) < len(data.index.names)) if include_index else False
    )
    if non_unique_cols or non_unique_indices:
        msg = (
            "Pandas dataframe contains non-unique indices and/or column names. "
            "Polars dataframes require unique string names for columns."
        )
        raise ValueError(msg)

    overlapping_cols_and_indices: set[str] = stringified_cols & stringified_index
    if len(overlapping_cols_and_indices) > 0:
        msg = "Pandas indices and column names must not overlap."
        raise ValueError(msg)


def pandas_to_pydf(
    data: pd.DataFrame,
    schema: SchemaDefinition | None = None,
    *,
    schema_overrides: SchemaDict | None = None,
    strict: bool = True,
    rechunk: bool = True,
    nan_to_null: bool = True,
    include_index: bool = False,
) -> PyDataFrame:
    """Construct a PyDataFrame from a pandas DataFrame."""
    _check_pandas_columns(data, include_index=include_index)

    convert_index = include_index and not _pandas_has_default_index(data)

    if not convert_index:

View on GitHub (pinned to df599052da)

Solutions

1. Rename the axis before converting: `df = df.rename_axis("a_idx")`
2. Or remove the colliding column: `df = df.drop(columns=["a"])` if the index already carries that data
3. Use `df.reset_index(drop=True)` when the index holds nothing you need, then convert without include_index
4. Convert without include_index (the default) so index names are never considered

Example fix

# before
import pandas as pd, polars as pl
pdf = pd.DataFrame({"a": [1, 2], "b": [3, 4]}).set_index("a", drop=False)
pl.from_pandas(pdf, include_index=True)  # ValueError: Pandas indices and column names must not overlap.

# after
pl.from_pandas(pdf.rename_axis("a_idx"), include_index=True)
# or drop the duplicate column:
pl.from_pandas(pdf.drop(columns=["a"]), include_index=True)

import pandas as pd

def index_cols_disjoint(df: pd.DataFrame) -> bool:
    idx = {str(n) for n in df.index.names}
    cols = {str(c) for c in df.columns}
    return not (idx & cols)

assert index_cols_disjoint(df), "index level name collides with a column name"
# then safe: pl.from_pandas(df, include_index=True)

import pandas as pd

def index_cols_disjoint(df: pd.DataFrame) -> bool:
    """True when no stringified index-level name equals a column name."""
    return not ({str(n) for n in df.index.names} & {str(c) for c in df.columns})

try:
    pldf = pl.from_pandas(df, include_index=True)
except ValueError as e:
    if "overlap" not in str(e):
        raise
    pldf = pl.from_pandas(df.rename_axis("_idx"), include_index=True)

When it happens

Trigger: `pl.from_pandas(df.set_index("a"), include_index=True)` while a column 'a' also still exists (set_index without dropping duplicates); any index level whose name (including default names like 'index' or None->stringified) collides with a data column name.

Common situations: set_index on a column that also remains in the frame (dup='keep' style patterns); reset_index followed by partial set_index roundtrips; Polars->pandas->Polars roundtrips where the pandas index inherited the column's name; default index name 'index' colliding with a real 'index' column.

Related errors

• Pandas dataframe contains non-unique indices and/or column n
• the given column-schema names do not match the data dictiona
• `orient` must be one of {'col', 'row', None}, got {orient!r}
• pyarrow is required for converting a pandas dataframe to Pol
• duplicate column names found: {series.columns.tolist()!s}

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/fdbf0532a36144d9. Report an issue: GitHub.