pola-rs/polars · error · ValueError

`columns` arg should only have unique values, got {columns!r

Error message

`columns` arg should only have unique values, got {columns!r}

What it means

After parse_columns_arg accepts a str or int sequence, _ensure_columns_are_unique (py-polars/src/polars/io/_utils.py:81-86) rejects duplicates because these readers cannot project the same column twice. Any repeated element in the list raises ValueError with the offending list embedded via {columns!r}.

Source

Thrown at py-polars/src/polars/io/_utils.py:84

    elif isinstance(columns, int):
        projection = [columns]
    elif is_str_sequence(columns):
        _ensure_columns_are_unique(columns)
        column_names = columns
    elif is_int_sequence(columns):
        _ensure_columns_are_unique(columns)
        projection = columns
    else:
        msg = "the `columns` argument should contain a list of all integers or all string values"
        raise TypeError(msg)

    return projection, column_names


def _ensure_columns_are_unique(columns: Sequence[str] | Sequence[int]) -> None:
    if len(columns) != len(set(columns)):
        msg = f"`columns` arg should only have unique values, got {columns!r}"
        raise ValueError(msg)


def parse_row_index_args(
    row_index_name: str | None = None,
    row_index_offset: int = 0,
) -> tuple[str, int] | None:
    """
    Parse the `row_index_name` and `row_index_offset` arguments of an I/O function.

    The Rust functions take a single tuple rather than two separate arguments.
    """
    if row_index_name is None:
        return None
    else:
        return (row_index_name, row_index_offset)


@overload

View on GitHub (pinned to df599052da)

Solutions

1. Deduplicate while preserving order: list(dict.fromkeys(columns))
2. Fix the upstream config or feature list that produced the duplicate
3. If the column is genuinely needed twice, read it once and copy it: df.with_columns(pl.col('a').alias('a_copy'))

Example fix

# before
df = pl.read_csv("f.csv", columns=["a", "b", "a"])
# after
df = pl.read_csv("f.csv", columns=list(dict.fromkeys(["a", "b", "a"])))

def unique_columns(columns):
    if len(columns) != len(set(columns)):
        dupes = {c for c in columns if columns.count(c) > 1}
        raise ValueError(f"duplicate columns requested: {dupes}")
    return columns

try:
    df = pl.read_csv(path, columns=cols)
except ValueError as e:
    if "unique values" in str(e):
        df = pl.read_csv(path, columns=list(dict.fromkeys(cols)))
    else:
        raise

When it happens

Trigger: pl.read_csv('f.csv', columns=['a','a']) or columns=[0,0]; a sequence built by concatenating two overlapping selections, e.g. base_cols + extra_cols where an entry repeats.

Common situations: Column lists assembled by concatenating feature lists that overlap; YAML/JSON pipeline configs with duplicated entries; refactors that merged index lists without deduping.

Related errors

• the `columns` argument should contain a list of all integers
• specified column names do not start with 'column_', but auto
• more dtypes overrides are specified than there are selected
• more new column names are specified than there are selected
• index positions should be smaller than 2^32

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/be1fe6c51975fbd0. Report an issue: GitHub.