{"record":{"id":"8376b5e2168cb5f4","repo":"pandas-dev/pandas","slug":"removals-must-all-be-in-old-categories-not-inclu","errorCode":null,"errorMessage":"removals must all be in old categories: {not_included}","messagePattern":"removals must all be in old categories: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"pandas/core/arrays/categorical.py","lineNumber":1492,"sourceCode":"        [NaN, 'c', 'b', 'c', NaN]\n        Categories (2, str): ['b', 'c']\n        \"\"\"\n        from pandas import Index\n\n        if not is_list_like(removals):\n            removals = [removals]\n\n        removals = Index(removals).unique().dropna()\n        new_categories = (\n            self.dtype.categories.difference(removals, sort=False)\n            if self.dtype.ordered is True\n            else self.dtype.categories.difference(removals)\n        )\n        not_included = removals.difference(self.dtype.categories)\n\n        if len(not_included) != 0:\n            not_included = set(not_included)\n            raise ValueError(f\"removals must all be in old categories: {not_included}\")\n\n        return self.set_categories(new_categories, ordered=self.ordered, rename=False)\n\n    def remove_unused_categories(self) -> Self:\n        \"\"\"\n        Remove categories which are not used.\n\n        This method is useful when working with datasets\n        that undergo dynamic changes where categories may no longer be\n        relevant, allowing to maintain a clean, efficient data structure.\n\n        Returns\n        -------\n        Categorical\n            Categorical with unused categories dropped.\n\n        See Also\n        --------","sourceCodeStart":1474,"sourceCodeEnd":1510,"githubUrl":"https://github.com/pandas-dev/pandas/blob/3b7651241d4da534b3559b60ef128e1c34f54116/pandas/core/arrays/categorical.py#L1474-L1510","documentation":"Raised by `remove_categories` when `removals` contains values that are not present in the current categories. Removing a non-existent label is treated as a programming error (likely a stale list) and is refused, with the offending entries reported via `not_included`.","triggerScenarios":"`cat.remove_categories(['x'])` where `'x'` is not in `cat.categories`; passing a list built from a different column or schema.","commonSituations":"Schema drift between runs; idempotent re-runs of a cleanup script; merging removal lists across differently-categorized DataFrames; refactoring that renamed labels upstream.","solutions":["Intersect the removal list with current categories: `to_remove = [r for r in removals if r in set(cat.categories)]; cat.remove_categories(to_remove)`.","Use `set_categories` with the desired final set to sidestep the existence check.","Verify each removal is in `cat.categories` before the call.","Make cleanup scripts idempotent by filtering removals through current categories."],"exampleFix":"# before\ncat = pd.Categorical(['a', 'b'], categories=['a', 'b', 'c'])\ncat = cat.remove_categories(['c', 'x'])  # ValueError: not_included={'x'}\n\n# after\nremovals = ['c', 'x']\nto_remove = [r for r in removals if r in cat.categories]\ncat = cat.remove_categories(to_remove)","handlingStrategy":"validation","validationCode":"def safe_remove_categories(cat, removals):\n    if not is_list_like(removals):\n        removals = [removals]\n    existing = set(cat.categories)\n    to_remove = [r for r in removals if r in existing]\n    return cat.remove_categories(to_remove) if to_remove else cat","typeGuard":"def are_all_present(cat, removals) -> bool:\n    existing = set(cat.categories)\n    return all(r in existing for r in removals)","tryCatchPattern":"try:\n    cat = cat.remove_categories(removals)\nexcept ValueError as e:\n    if 'must all be in old categories' in str(e):\n        cat = cat.remove_categories([r for r in removals if r in set(cat.categories)])\n    else:\n        raise","preventionTips":["Intersect `removals` with current categories before `remove_categories` for idempotent cleanups.","Use `set_categories` with the desired final set to bypass existence checks.","Validate each removal against `cat.categories` before the call."],"tags":["categorical","remove-categories","valueerror"],"backgroundTag":null,"analyzedSha":"3b7651241d4da534b3559b60ef128e1c34f54116","analyzedAt":"2026-08-11T22:10:44.015Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}