pandas-dev/pandas · error · ValueError

values should be unique if codes is not None

Error message

values should be unique if codes is not None

What it means

Raised by safe_sort on the fast counting-sort path: when codes is not None and assume_unique is False, pandas verifies values are unique by comparing the counting-sort coverage to len(values). If values contain duplicates the code remap would be ambiguous, so a ValueError is raised. The message notes uniqueness is required only when codes is provided.

Source

Thrown at pandas/core/algorithms.py:1752

    codes = ensure_platform_int(np.asarray(codes))

    # ranks[i] gives the position of values[i] in `ordered`
    if use_counting:
        arr = cast("np.ndarray", values)
        if arr.dtype.kind == "i":
            # go through int64 so differences don't overflow narrower signed
            #  dtypes; int64 wraparound is exact since the true differences
            #  are within rng_size
            shifted = arr.astype(np.int64, copy=False) - vmin
        else:
            # unsigned: differences always fit the unsigned dtype
            shifted = arr - vmin
        present = np.zeros(rng_size, dtype=bool)
        present[shifted] = True
        counts = present.cumsum(dtype=np.intp)
        # the counting pass gives the uniqueness check for free
        if not assume_unique and counts[-1] != len(values):
            raise ValueError("values should be unique if codes is not None")
        ranks = counts[shifted]
        ranks -= 1
    else:
        if not assume_unique and not len(unique(values)) == len(values):
            raise ValueError("values should be unique if codes is not None")

        if sorter is not None:
            # sorter is a permutation, so scatter is a faster equivalent to
            #  `ranks = sorter.argsort()`
            ranks = np.empty(len(sorter), dtype=np.intp)
            ranks[sorter] = np.arange(len(sorter), dtype=np.intp)
        else:
            # mixed types
            # error: Argument 1 to "_get_hashtable_algo" has incompatible type
            # "Union[Index, ExtensionArray, ndarray[Any, Any]]"; expected
            # "ndarray[Any, Any]"
            hash_klass, values = _get_hashtable_algo(values)  # type: ignore[arg-type]
            t = hash_klass(len(values))

View on GitHub (pinned to 71959b8cb9)

Solutions

1. Deduplicate values before calling safe_sort (e.g. np.unique).
2. If values really are unique, pass assume_unique=True to skip the check (counting path).
3. Pass codes=None if you only need the sorted values.

Example fix

# before
safe_sort(np.array([1, 1, 2, 3]), codes=[0, 1, 2])
# after
vals, idx = np.unique([1, 1, 2, 3], return_index=True)
safe_sort(vals, codes=idx, assume_unique=True)

import numpy as np

def safe_sort_unique(values, codes, assume_unique=False):
    if codes is not None and not assume_unique:
        if len(np.unique(values)) != len(values):
            values, inv = np.unique(values, return_inverse=True)
            codes = inv[np.asarray(codes)]
            assume_unique = True
    return pd.core.algorithms.safe_sort(values, codes, assume_unique=assume_unique)

import numpy as np

def values_are_unique(values) -> bool:
    v = np.asarray(values)
    return len(np.unique(v)) == len(v)

When it happens

Trigger: safe_sort(np.array([1, 1, 2]), codes=[0, 1, 2]) on an integer array large enough to trigger counting sort, with duplicate values present; factorize/unique pipelines that feed duplicate values plus codes into safe_sort.

Common situations: Pre-aggregated data where the 'values' index accidentally contains repeats; merging arrays that introduced duplicates before sorting; passing assume_unique=False (default) on data that still has dups.

Related errors

• Only list-like objects or None are allowed to be passed to s
• Only np.ndarray, ExtensionArray, and Index objects are allow
• No such keys(s): {pat!r}
• {k} is not a valid identifier
• {k} is a python keyword

AI-assisted analysis of pandas-dev/pandas@71959b8cb9 (2026-08-07). Data as JSON: /api/errors/eaede111f49e5ba4. Report an issue: GitHub.