{"record":{"id":"eee877f216c02cbd","repo":"pola-rs/polars","slug":"pandas-dataframe-contains-non-unique-indices-and-o","errorCode":null,"errorMessage":"Pandas dataframe contains non-unique indices and/or column names. Polars dataframes require unique string names for columns.","messagePattern":"Pandas dataframe contains non-unique indices and/or column names\\. Polars dataframes require unique string names for columns\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"py-polars/src/polars/_utils/construction/dataframe.py","lineNumber":1077,"sourceCode":"\n\ndef _check_pandas_columns(data: pd.DataFrame, *, include_index: bool) -> None:\n    \"\"\"Check pandas dataframe columns can be converted to polars.\"\"\"\n    stringified_cols: set[str] = {str(col) for col in data.columns}\n    stringified_index: set[str] = (\n        {str(idx) for idx in data.index.names} if include_index else set()\n    )\n\n    non_unique_cols: bool = len(stringified_cols) < len(data.columns)\n    non_unique_indices: bool = (\n        (len(stringified_index) < len(data.index.names)) if include_index else False\n    )\n    if non_unique_cols or non_unique_indices:\n        msg = (\n            \"Pandas dataframe contains non-unique indices and/or column names. \"\n            \"Polars dataframes require unique string names for columns.\"\n        )\n        raise ValueError(msg)\n\n    overlapping_cols_and_indices: set[str] = stringified_cols & stringified_index\n    if len(overlapping_cols_and_indices) > 0:\n        msg = \"Pandas indices and column names must not overlap.\"\n        raise ValueError(msg)\n\n\ndef pandas_to_pydf(\n    data: pd.DataFrame,\n    schema: SchemaDefinition | None = None,\n    *,\n    schema_overrides: SchemaDict | None = None,\n    strict: bool = True,\n    rechunk: bool = True,\n    nan_to_null: bool = True,\n    include_index: bool = False,\n) -> PyDataFrame:\n    \"\"\"Construct a PyDataFrame from a pandas DataFrame.\"\"\"","sourceCodeStart":1059,"sourceCodeEnd":1095,"githubUrl":"https://github.com/pola-rs/polars/blob/df599052daf96e7a9cc30a3b0c6bd25d6947e3c0/py-polars/src/polars/_utils/construction/dataframe.py#L1059-L1095","documentation":"Before converting a pandas DataFrame, polars stringifies column names (and, with include_index, index-level names) and requires them to be unique — Polars identifies columns by unique string names. If stringification collapses entries (duplicate labels like two 'a' columns, or mixed types that stringify equal, e.g. int 1 and str '1'), this ValueError is raised instead of producing an invalid schema.","triggerScenarios":"`pl.from_pandas(df)` where df has duplicate column labels (common after joins/aggregations without suffixes or read_csv with repeated headers); or include_index conversion where two index levels stringify to the same name; columns whose names are distinct in pandas but equal after str().","commonSituations":"pandas joins without lsuffix/rsuffix producing duplicated names; groupby/agg that keeps the grouping key plus a same-named aggregate; MultiIndex flattened to repeated level names; numeric column labels 0..n alongside string '0'.","solutions":["Deduplicate before converting: `df = df.loc[:, ~df.columns.duplicated()]` (keeps first occurrence)","Rename to unique names: `df.columns = [f\"{c}_{i}\" if duplicated else c ...]` or `df.add_suffix(\"_2\")` where appropriate","Fix the upstream pandas operation (named aggregation, join with lsuffix/rsuffix) so duplicates never appear","For MultiIndex, flatten first: `df.columns = [\"_\".join(map(str, lvl)) for lvl in df.columns]`"],"exampleFix":"# before\nimport pandas as pd, polars as pl\npdf = pd.DataFrame([[1, 2]], columns=[\"a\", \"a\"])\npl.from_pandas(pdf)  # ValueError: Pandas dataframe contains non-unique indices and/or column names ...\n\n# after\npdf = pdf.loc[:, ~pdf.columns.duplicated()]\n# or make unique explicitly:\npdf.columns = [\"a\", \"a_2\"]\npl.from_pandas(pdf)","handlingStrategy":"validation","validationCode":"import pandas as pd\n\nassert df.columns.is_unique, f\"duplicate pandas columns: {df.columns[df.columns.duplicated()].tolist()}\"\nassert df.index.names == [None] or len(set(map(str, df.index.names))) == len(df.index.names)\npl.from_pandas(df)","typeGuard":"import pandas as pd\n\ndef pandas_cols_unique(df: pd.DataFrame) -> bool:\n    \"\"\"True if stringified column names (and index names) are unique.\"\"\"\n    cols_ok = len({str(c) for c in df.columns}) == len(df.columns)\n    idx_ok = len({str(n) for n in df.index.names}) == len(df.index.names)\n    return cols_ok and idx_ok","tryCatchPattern":"try:\n    pl.from_pandas(df)\nexcept ValueError as e:\n    if \"non-unique\" not in str(e):\n        raise\n    df = df.loc[:, ~df.columns.duplicated()]\n    pl.from_pandas(df)","preventionTips":["Assert df.columns.is_unique before conversion in ingestion helpers","Use lsuffix/rsuffix on pandas joins and named aggregations so duplicates never occur","Flatten MultiIndex columns to underscore-joined strings before converting"],"tags":["pandas-interop","duplicate-columns","dataframe-construction","validation"],"backgroundTag":null,"analyzedSha":"df599052daf96e7a9cc30a3b0c6bd25d6947e3c0","analyzedAt":"2026-08-16T12:10:03.978Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}