{"record":{"id":"781d8cbaeb1e16cd","repo":"pandas-dev/pandas","slug":"new-categories-must-not-include-old-categories-a","errorCode":null,"errorMessage":"new categories must not include old categories: {already_included}","messagePattern":"new categories must not include old categories: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"pandas/core/arrays/categorical.py","lineNumber":1414,"sourceCode":"        set_categories : Set the categories to the specified ones.\n\n        Examples\n        --------\n        >>> c = pd.Categorical([\"c\", \"b\", \"c\"])\n        >>> c\n        ['c', 'b', 'c']\n        Categories (2, str): ['b', 'c']\n\n        >>> c.add_categories([\"d\", \"a\"])\n        ['c', 'b', 'c']\n        Categories (4, str): ['b', 'c', 'd', 'a']\n        \"\"\"\n\n        if not is_list_like(new_categories):\n            new_categories = [new_categories]\n        already_included = set(new_categories) & set(self.dtype.categories)\n        if len(already_included) != 0:\n            raise ValueError(\n                f\"new categories must not include old categories: {already_included}\"\n            )\n\n        if hasattr(new_categories, \"dtype\"):\n            from pandas import Series\n\n            dtype = find_common_type(\n                [self.dtype.categories.dtype, new_categories.dtype]\n            )\n            new_categories = Series(\n                list(self.dtype.categories) + list(new_categories), dtype=dtype\n            )\n        else:\n            new_categories = list(self.dtype.categories) + list(new_categories)\n\n        new_dtype = CategoricalDtype(new_categories, self.ordered)\n        cat = self.copy()\n        codes = coerce_indexer_dtype(cat._ndarray, new_dtype.categories)","sourceCodeStart":1396,"sourceCodeEnd":1432,"githubUrl":"https://github.com/pandas-dev/pandas/blob/71959b8cb9b2459c16e14b34f28b178ccfe14735/pandas/core/arrays/categorical.py#L1396-L1432","documentation":"Raised by add_categories when one or more of the supplied new_categories already exist in the current categories. Adding a duplicate category is meaningless and would corrupt the code-to-category mapping, so pandas reports the offending set in the message.","triggerScenarios":"`cat.add_categories(['a'])` where 'a' is already a category; or `cat.add_categories(['x','y'])` where 'y' already exists.","commonSituations":"Building the union of categories by appending a list that overlaps the existing ones; or appending 'other'/'unknown' repeatedly across pipeline stages.","solutions":["Filter out existing categories first: `cat.add_categories([c for c in new if c not in cat.categories])`.","Use set logic: `cat.add_categories(set(new) - set(cat.categories))`.","If you intended to replace, use `set_categories` instead."],"exampleFix":"# before\ncat = pd.Categorical(['a','b'], categories=['a','b'])\ncat.add_categories(['a','c'])\n# after\ncat = pd.Categorical(['a','b'], categories=['a','b'])\ncat.add_categories(['c'])","handlingStrategy":"validation","validationCode":"def safe_add_categories(cat, new_categories):\n    dup = set(new_categories) & set(cat.categories)\n    if dup:\n        raise ValueError(f\"already categories: {dup}\")\n    return cat.add_categories(new_categories)","typeGuard":"def all_new_categories(cat, new_categories) -> bool:\n    return not (set(new_categories) & set(cat.categories))","tryCatchPattern":"try:\n    cat = cat.add_categories(new_cats)\nexcept ValueError as e:\n    if 'must not include old categories' in str(e):\n        cat = cat.add_categories([c for c in new_cats if c not in cat.categories])\n    else:\n        raise","preventionTips":["Filter new categories against the current set before adding.","Use set differences to compute the novel categories.","Replace set_categories if you intend a full replacement."],"tags":["categorical","add-categories","duplicate","valueerror"],"analyzedSha":"71959b8cb9b2459c16e14b34f28b178ccfe14735","analyzedAt":"2026-08-07T01:30:20.476Z","schemaVersion":2},"datasetVersion":"2026-08-07T03:17:09.362Z"}