{"record":{"id":"be1fe6c51975fbd0","repo":"pola-rs/polars","slug":"columns-arg-should-only-have-unique-values-got","errorCode":null,"errorMessage":"`columns` arg should only have unique values, got {columns!r}","messagePattern":"`columns` arg should only have unique values, got (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"py-polars/src/polars/io/_utils.py","lineNumber":84,"sourceCode":"    elif isinstance(columns, int):\n        projection = [columns]\n    elif is_str_sequence(columns):\n        _ensure_columns_are_unique(columns)\n        column_names = columns\n    elif is_int_sequence(columns):\n        _ensure_columns_are_unique(columns)\n        projection = columns\n    else:\n        msg = \"the `columns` argument should contain a list of all integers or all string values\"\n        raise TypeError(msg)\n\n    return projection, column_names\n\n\ndef _ensure_columns_are_unique(columns: Sequence[str] | Sequence[int]) -> None:\n    if len(columns) != len(set(columns)):\n        msg = f\"`columns` arg should only have unique values, got {columns!r}\"\n        raise ValueError(msg)\n\n\ndef parse_row_index_args(\n    row_index_name: str | None = None,\n    row_index_offset: int = 0,\n) -> tuple[str, int] | None:\n    \"\"\"\n    Parse the `row_index_name` and `row_index_offset` arguments of an I/O function.\n\n    The Rust functions take a single tuple rather than two separate arguments.\n    \"\"\"\n    if row_index_name is None:\n        return None\n    else:\n        return (row_index_name, row_index_offset)\n\n\n@overload","sourceCodeStart":66,"sourceCodeEnd":102,"githubUrl":"https://github.com/pola-rs/polars/blob/df599052daf96e7a9cc30a3b0c6bd25d6947e3c0/py-polars/src/polars/io/_utils.py#L66-L102","documentation":"After parse_columns_arg accepts a str or int sequence, _ensure_columns_are_unique (py-polars/src/polars/io/_utils.py:81-86) rejects duplicates because these readers cannot project the same column twice. Any repeated element in the list raises ValueError with the offending list embedded via {columns!r}.","triggerScenarios":"pl.read_csv('f.csv', columns=['a','a']) or columns=[0,0]; a sequence built by concatenating two overlapping selections, e.g. base_cols + extra_cols where an entry repeats.","commonSituations":"Column lists assembled by concatenating feature lists that overlap; YAML/JSON pipeline configs with duplicated entries; refactors that merged index lists without deduping.","solutions":["Deduplicate while preserving order: list(dict.fromkeys(columns))","Fix the upstream config or feature list that produced the duplicate","If the column is genuinely needed twice, read it once and copy it: df.with_columns(pl.col('a').alias('a_copy'))"],"exampleFix":"# before\ndf = pl.read_csv(\"f.csv\", columns=[\"a\", \"b\", \"a\"])\n# after\ndf = pl.read_csv(\"f.csv\", columns=list(dict.fromkeys([\"a\", \"b\", \"a\"])))","handlingStrategy":"validation","validationCode":"def unique_columns(columns):\n    if len(columns) != len(set(columns)):\n        dupes = {c for c in columns if columns.count(c) > 1}\n        raise ValueError(f\"duplicate columns requested: {dupes}\")\n    return columns","typeGuard":null,"tryCatchPattern":"try:\n    df = pl.read_csv(path, columns=cols)\nexcept ValueError as e:\n    if \"unique values\" in str(e):\n        df = pl.read_csv(path, columns=list(dict.fromkeys(cols)))\n    else:\n        raise","preventionTips":["Normalize any user/config-provided column list through list(dict.fromkeys(...)) before use","Assert uniqueness in config loaders with a clear error message","Lint ETL configs for duplicated column entries"],"tags":["polars","io","columns","duplicates","valueerror"],"backgroundTag":null,"analyzedSha":"df599052daf96e7a9cc30a3b0c6bd25d6947e3c0","analyzedAt":"2026-08-16T12:10:03.978Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}