{"record":{"id":"781d8cbaeb1e16cd","repo":"pandas-dev/pandas","slug":"new-categories-must-not-include-old-categories-a","errorCode":null,"errorMessage":"new categories must not include old categories: {already_included}","messagePattern":"new categories must not include old categories: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"pandas/core/arrays/categorical.py","lineNumber":1414,"sourceCode":"        set_categories : Set the categories to the specified ones.\n\n        Examples\n        --------\n        >>> c = pd.Categorical([\"c\", \"b\", \"c\"])\n        >>> c\n        ['c', 'b', 'c']\n        Categories (2, str): ['b', 'c']\n\n        >>> c.add_categories([\"d\", \"a\"])\n        ['c', 'b', 'c']\n        Categories (4, str): ['b', 'c', 'd', 'a']\n        \"\"\"\n\n        if not is_list_like(new_categories):\n            new_categories = [new_categories]\n        already_included = set(new_categories) & set(self.dtype.categories)\n        if len(already_included) != 0:\n            raise ValueError(\n                f\"new categories must not include old categories: {already_included}\"\n            )\n\n        if hasattr(new_categories, \"dtype\"):\n            from pandas import Series\n\n            dtype = find_common_type(\n                [self.dtype.categories.dtype, new_categories.dtype]\n            )\n            new_categories = Series(\n                list(self.dtype.categories) + list(new_categories), dtype=dtype\n            )\n        else:\n            new_categories = list(self.dtype.categories) + list(new_categories)\n\n        new_dtype = CategoricalDtype(new_categories, self.ordered)\n        cat = self.copy()\n        codes = coerce_indexer_dtype(cat._ndarray, new_dtype.categories)","sourceCodeStart":1396,"sourceCodeEnd":1432,"githubUrl":"https://github.com/pandas-dev/pandas/blob/3b7651241d4da534b3559b60ef128e1c34f54116/pandas/core/arrays/categorical.py#L1396-L1432","documentation":"Raised by `add_categories` when any value in `new_categories` is already present in the current categories. Adding a duplicate would create ambiguous codes and silently inflate cardinality, so pandas rejects the call and reports the offending intersection.","triggerScenarios":"`cat.add_categories(['a'])` where `'a'` is already in `cat.categories`, or `add_categories(['d', 'a'])` where `'a'` already exists.","commonSituations":"Idempotent re-application of category additions; merging category lists from multiple sources without deduping; loops that add categories one-by-one and re-process.","solutions":["Dedupe against existing categories first: `to_add = [c for c in new if c not in set(cat.categories)]; cat.add_categories(to_add)`.","Use `set_categories` with the union if you want to force a final category set.","Wrap repeated additions in an `if c not in cat.categories` guard.","Make the operation idempotent: filter out `already_included` before calling."],"exampleFix":"# before\ncat = pd.Categorical(['b', 'c'], categories=['b', 'c'])\ncat = cat.add_categories(['d', 'b'])  # ValueError: already_included={'b'}\n\n# after\nnew = ['d', 'b']\nto_add = [c for c in new if c not in cat.categories]\ncat = cat.add_categories(to_add)","handlingStrategy":"validation","validationCode":"def safe_add_categories(cat, new_categories):\n    if not is_list_like(new_categories):\n        new_categories = [new_categories]\n    existing = set(cat.categories)\n    to_add = [c for c in new_categories if c not in existing]\n    return cat.add_categories(to_add) if to_add else cat","typeGuard":"def are_all_new(cat, new_categories) -> bool:\n    existing = set(cat.categories)\n    return all(c not in existing for c in new_categories)","tryCatchPattern":"try:\n    cat = cat.add_categories(new)\nexcept ValueError as e:\n    if 'must not include old categories' in str(e):\n        cat = cat.add_categories([c for c in new if c not in set(cat.categories)])\n    else:\n        raise","preventionTips":["Filter out already-present categories before `add_categories` to make additions idempotent.","When merging category lists, dedupe against the existing set.","Use `set_categories` with the union if you want a single declarative update."],"tags":["categorical","add-categories","duplicate","valueerror"],"backgroundTag":null,"analyzedSha":"3b7651241d4da534b3559b60ef128e1c34f54116","analyzedAt":"2026-08-11T22:10:44.015Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}