pandas-dev/pandas · error · NotImplementedError

unable to collapse Joint Filters

Error message

unable to collapse Joint Filters

What it means

Raised in JointFilterBinOp.format. JointFilterBinOp represents a boolean combination (AND/OR) of two FilterBinOp terms; unlike conditions (which can be collapsed into a numexpr string), filters cannot be merged into a single filter tuple because each side carries its own (column, op, Index) triple. Calling format() on such a joint node is unsupported, so NotImplementedError is raised to signal that the caller must evaluate each side separately.

Solutions

1. Reduce the size of 'in' lists so at least one side takes the condition path (<= _max_selectors, default 31).
2. Split the query into two selects and intersect coordinates: c1 = store.select_as_coordinates('df', 'A in [1,2]'); c2 = store.select_as_coordinates('df', 'B in [3,4]'); store.select('df', where=c1.intersection(c2)).
3. Read the data and filter in pandas with isin on both columns.
4. Precompute a combined data_column to avoid joint filters.

Example fix

// before
store.select('df', where='(A in huge1) & (B in huge2)')

// after
c1 = store.select_as_coordinates('df', 'A in huge1')
c2 = store.select_as_coordinates('df', 'B in huge2')
store.select('df', where=c1.intersection(c2))

def joint_filter_safe(store, key: str, list_sizes: list[int]) -> bool:
    storer = store.get_storer(key)
    max_sel = getattr(storer, '_max_selectors', 31)
    return sum(1 for n in list_sizes if n > max_sel) <= 1

assert joint_filter_safe(store, 'df', [len(a), len(b)])

def at_most_one_filter_branch(list_sizes: list[int], max_selectors: int = 31) -> bool:
    return sum(1 for n in list_sizes if n > max_selectors) <= 1

try:
    store.select('df', where='(A in huge1) & (B in huge2)')
except NotImplementedError as e:
    if 'Joint Filters' in str(e):
        c1 = store.select_as_coordinates('df', 'A in huge1')
        c2 = store.select_as_coordinates('df', 'B in huge2')
        store.select('df', where=c1.intersection(c2))
    else:
        raise

When it happens

Trigger: store.select('df', where='(A in [1,2]) & (B in [3,4])') where both sides exceed _max_selectors and become FilterBinOps, then the combination is a JointFilterBinOp whose format() is invoked during result assembly. Also triggered by chaining multiple value-list filters with & / |.

Common situations: Large 'in' lists on multiple columns combined with boolean operators. Workloads ported from SQL IN ... AND IN ... patterns. Misconfigured _max_selectors causing both branches to take the filter path.

Related errors

• passing a filterable condition to a non-table indexer
• query term is not valid
• arithmetic operations are not supported inside an HDFStore…
• Cannot compare of type to column
• cannot subscript with

AI-assisted analysis of pandas-dev/pandas@3b7651241d (2026-08-11). Data as JSON: /api/errors/3fddefe4f6438637. Report an issue: GitHub.

Appendix: source

Thrown at pandas/core/computation/pytables.py:367

                )
            return None

        if self.is_in_table and len(values) <= self._max_selectors:
            return None
        filter_op = self.generate_filter_op()
        self.filter = (self.lhs.value, filter_op, Index(values))
        return self

    def generate_filter_op(self, invert: bool = False):
        if (self.op == "!=" and not invert) or (self.op == "==" and invert):
            return lambda axis, vals: ~axis.isin(vals)
        else:
            return lambda axis, vals: axis.isin(vals)


class JointFilterBinOp(FilterBinOp):
    def format(self):
        raise NotImplementedError("unable to collapse Joint Filters")

    # error: Signature of "evaluate" incompatible with supertype "BinOp"
    def evaluate(self) -> Self:  # type: ignore[override]
        return self


class ConditionBinOp(BinOp):
    def __repr__(self) -> str:
        return pprint_thing(f"[Condition : [{self.condition}]]")

    def invert(self):
        """invert the condition"""
        # if self.condition is not None:
        #    self.condition = "~(%s)" % self.condition
        # return self
        raise NotImplementedError(
            "cannot use an invert condition when passing to numexpr"
        )

View on GitHub (pinned to 3b7651241d)