{"record":{"id":"3fddefe4f6438637","repo":"pandas-dev/pandas","slug":"unable-to-collapse-joint-filters","errorCode":null,"errorMessage":"unable to collapse Joint Filters","messagePattern":"unable to collapse Joint Filters","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"pandas/core/computation/pytables.py","lineNumber":367,"sourceCode":"                )\n            return None\n\n        if self.is_in_table and len(values) <= self._max_selectors:\n            return None\n        filter_op = self.generate_filter_op()\n        self.filter = (self.lhs.value, filter_op, Index(values))\n        return self\n\n    def generate_filter_op(self, invert: bool = False):\n        if (self.op == \"!=\" and not invert) or (self.op == \"==\" and invert):\n            return lambda axis, vals: ~axis.isin(vals)\n        else:\n            return lambda axis, vals: axis.isin(vals)\n\n\nclass JointFilterBinOp(FilterBinOp):\n    def format(self):\n        raise NotImplementedError(\"unable to collapse Joint Filters\")\n\n    # error: Signature of \"evaluate\" incompatible with supertype \"BinOp\"\n    def evaluate(self) -> Self:  # type: ignore[override]\n        return self\n\n\nclass ConditionBinOp(BinOp):\n    def __repr__(self) -> str:\n        return pprint_thing(f\"[Condition : [{self.condition}]]\")\n\n    def invert(self):\n        \"\"\"invert the condition\"\"\"\n        # if self.condition is not None:\n        #    self.condition = \"~(%s)\" % self.condition\n        # return self\n        raise NotImplementedError(\n            \"cannot use an invert condition when passing to numexpr\"\n        )","sourceCodeStart":349,"sourceCodeEnd":385,"githubUrl":"https://github.com/pandas-dev/pandas/blob/3b7651241d4da534b3559b60ef128e1c34f54116/pandas/core/computation/pytables.py#L349-L385","documentation":"Raised in JointFilterBinOp.format. JointFilterBinOp represents a boolean combination (AND/OR) of two FilterBinOp terms; unlike conditions (which can be collapsed into a numexpr string), filters cannot be merged into a single filter tuple because each side carries its own (column, op, Index) triple. Calling format() on such a joint node is unsupported, so NotImplementedError is raised to signal that the caller must evaluate each side separately.","triggerScenarios":"store.select('df', where='(A in [1,2]) & (B in [3,4])') where both sides exceed _max_selectors and become FilterBinOps, then the combination is a JointFilterBinOp whose format() is invoked during result assembly. Also triggered by chaining multiple value-list filters with & / |.","commonSituations":"Large 'in' lists on multiple columns combined with boolean operators. Workloads ported from SQL IN ... AND IN ... patterns. Misconfigured _max_selectors causing both branches to take the filter path.","solutions":["Reduce the size of 'in' lists so at least one side takes the condition path (<= _max_selectors, default 31).","Split the query into two selects and intersect coordinates: c1 = store.select_as_coordinates('df', 'A in [1,2]'); c2 = store.select_as_coordinates('df', 'B in [3,4]'); store.select('df', where=c1.intersection(c2)).","Read the data and filter in pandas with isin on both columns.","Precompute a combined data_column to avoid joint filters."],"exampleFix":"// before\nstore.select('df', where='(A in huge1) & (B in huge2)')\n\n// after\nc1 = store.select_as_coordinates('df', 'A in huge1')\nc2 = store.select_as_coordinates('df', 'B in huge2')\nstore.select('df', where=c1.intersection(c2))","handlingStrategy":"fallback","validationCode":"def joint_filter_safe(store, key: str, list_sizes: list[int]) -> bool:\n    storer = store.get_storer(key)\n    max_sel = getattr(storer, '_max_selectors', 31)\n    return sum(1 for n in list_sizes if n > max_sel) <= 1\n\nassert joint_filter_safe(store, 'df', [len(a), len(b)])","typeGuard":"def at_most_one_filter_branch(list_sizes: list[int], max_selectors: int = 31) -> bool:\n    return sum(1 for n in list_sizes if n > max_selectors) <= 1","tryCatchPattern":"try:\n    store.select('df', where='(A in huge1) & (B in huge2)')\nexcept NotImplementedError as e:\n    if 'Joint Filters' in str(e):\n        c1 = store.select_as_coordinates('df', 'A in huge1')\n        c2 = store.select_as_coordinates('df', 'B in huge2')\n        store.select('df', where=c1.intersection(c2))\n    else:\n        raise","preventionTips":["Keep at most one 'in' list above _max_selectors per where clause.","Intersect coordinates from separate selects instead of one joint filter.","Precompute a combined data_column when joint filtering is frequent."],"tags":["pandas","hdfstore","pytables","filter","joint-filter","where"],"backgroundTag":null,"analyzedSha":"3b7651241d4da534b3559b60ef128e1c34f54116","analyzedAt":"2026-08-11T22:10:44.015Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}