pandas-dev/pandas · error · ValueError

new categories must not include old categories: {already_inc

Error message

new categories must not include old categories: {already_included}

What it means

Raised by add_categories when one or more of the supplied new_categories already exist in the current categories. Adding a duplicate category is meaningless and would corrupt the code-to-category mapping, so pandas reports the offending set in the message.

Source

Thrown at pandas/core/arrays/categorical.py:1414

        set_categories : Set the categories to the specified ones.

        Examples
        --------
        >>> c = pd.Categorical(["c", "b", "c"])
        >>> c
        ['c', 'b', 'c']
        Categories (2, str): ['b', 'c']

        >>> c.add_categories(["d", "a"])
        ['c', 'b', 'c']
        Categories (4, str): ['b', 'c', 'd', 'a']
        """

        if not is_list_like(new_categories):
            new_categories = [new_categories]
        already_included = set(new_categories) & set(self.dtype.categories)
        if len(already_included) != 0:
            raise ValueError(
                f"new categories must not include old categories: {already_included}"
            )

        if hasattr(new_categories, "dtype"):
            from pandas import Series

            dtype = find_common_type(
                [self.dtype.categories.dtype, new_categories.dtype]
            )
            new_categories = Series(
                list(self.dtype.categories) + list(new_categories), dtype=dtype
            )
        else:
            new_categories = list(self.dtype.categories) + list(new_categories)

        new_dtype = CategoricalDtype(new_categories, self.ordered)
        cat = self.copy()
        codes = coerce_indexer_dtype(cat._ndarray, new_dtype.categories)

View on GitHub (pinned to 71959b8cb9)

Solutions

1. Filter out existing categories first: `cat.add_categories([c for c in new if c not in cat.categories])`.
2. Use set logic: `cat.add_categories(set(new) - set(cat.categories))`.
3. If you intended to replace, use `set_categories` instead.

Example fix

# before
cat = pd.Categorical(['a','b'], categories=['a','b'])
cat.add_categories(['a','c'])
# after
cat = pd.Categorical(['a','b'], categories=['a','b'])
cat.add_categories(['c'])

def safe_add_categories(cat, new_categories):
    dup = set(new_categories) & set(cat.categories)
    if dup:
        raise ValueError(f"already categories: {dup}")
    return cat.add_categories(new_categories)

def all_new_categories(cat, new_categories) -> bool:
    return not (set(new_categories) & set(cat.categories))

try:
    cat = cat.add_categories(new_cats)
except ValueError as e:
    if 'must not include old categories' in str(e):
        cat = cat.add_categories([c for c in new_cats if c not in cat.categories])
    else:
        raise

When it happens

Trigger: `cat.add_categories(['a'])` where 'a' is already a category; or `cat.add_categories(['x','y'])` where 'y' already exists.

Common situations: Building the union of categories by appending a list that overlaps the existing ones; or appending 'other'/'unknown' repeatedly across pipeline stages.

Related errors

• Lengths must match.
• Cannot convert float NaN to integer
• Cannot cast {self.categories.dtype} dtype to {dtype}
• The categories must be provided in 'categories' or 'dtype'.
• new categories need to have the same number of items as the

AI-assisted analysis of pandas-dev/pandas@71959b8cb9 (2026-08-07). Data as JSON: /api/errors/781d8cbaeb1e16cd. Report an issue: GitHub.