microsoft/semantic-kernel · error · VectorStoreOperationException

Filter string exceeds the maximum allowed length.

Error message

Filter string exceeds the maximum allowed length.

What it means

Thrown by _parse_and_validate_filter (in_memory.py:750-751) when the raw filter string is longer than max_filter_source_length (default 2048 characters), before any parsing. It is the first guard against oversized filter input.

Solutions

  1. Shorten the filter string (e.g. move large literal sets out of the filter).
  2. Pass options.filter as a Python callable instead of a string to bypass the source-length limit for trusted logic.
  3. Raise max_filter_source_length on the collection instance if a genuinely long filter is required.
  4. Split a single long filter into a list of shorter filters (options.filter accepts a list, OR-semantics).

Example fix

# before
collection.max_filter_source_length = 2048  # default
opts = VectorSearchOptions(filter=f"lambda x: x.id in {str(list(range(5000)))}")  # huge
# after
allowed = set(range(5000))
opts = VectorSearchOptions(filter=lambda x: x.id in allowed)  # callable, no length limit
Defensive patterns

Strategy: validation

Validate before calling

def within_source_limit(expr: str, cap: int = 2048) -> bool:
    return len(expr) <= cap

Try / catch

try:
    results = await collection.search(search_type=SearchType.VECTOR, options=opts)
except VectorStoreOperationException as e:
    logger.warning("filter rejected: %s", e.__cause__ or e)
    results = None

Prevention

When it happens

Trigger: Passing a very long lambda string, typically one containing a huge inline literal collection or very long string constants.

Common situations: Inlining a large ID allowlist into the filter; generating filters from templates that balloon in size; lowering the limit on a shared collection.

Related errors

AI-assisted analysis of microsoft/semantic-kernel@c028a0c7dc (2026-08-13). Data as JSON: /api/errors/e696fdd0a9f25583. Report an issue: GitHub.

Appendix: source

Thrown at python/semantic_kernel/connectors/in_memory.py:751

            ]
        except Exception as e:
            raise VectorStoreOperationException(f"Error evaluating filter: {e}") from e
        filtered_records: dict[TKey, AttributeDict] = {}
        for key, record in self.inner_storage.items():
            for filter in callable_filters:
                if self._run_filter(filter, record):
                    filtered_records[key] = record
        return filtered_records

    def _parse_and_validate_filter(self, filter_str: str) -> Callable:
        """Parse and validate a string filter as a lambda expression, then return the callable.

        Uses an allowlist approach - only explicitly permitted AST node types and function names
        are allowed. This can be customized by overriding `allowed_filter_ast_nodes` and
        `allowed_filter_functions` class attributes.
        """
        if len(filter_str) > self.max_filter_source_length:
            raise VectorStoreOperationException("Filter string exceeds the maximum allowed length.")

        try:
            tree = ast.parse(filter_str, mode="eval")
        except SyntaxError as e:
            raise VectorStoreOperationException(f"Filter string is not valid Python: {e}") from e

        # Only allow lambda expressions at the top level
        if not (isinstance(tree, ast.Expression) and isinstance(tree.body, ast.Lambda)):
            raise VectorStoreOperationException(
                "Filter string must be a lambda expression, e.g. 'lambda x: x.key == 1'"
            )

        # Get the lambda parameter name(s) to allow them as valid Name nodes
        lambda_node = tree.body
        lambda_param_names = {arg.arg for arg in lambda_node.args.args}
        lambda_param_order = [arg.arg for arg in lambda_node.args.args]
        # Walk the AST to validate all nodes against the allowlist
        for node_count, node in enumerate(ast.walk(tree), start=1):

View on GitHub (pinned to c028a0c7dc)