{"record":{"id":"e696fdd0a9f25583","repo":"microsoft/semantic-kernel","slug":"filter-string-exceeds-the-maximum-allowed-length","errorCode":null,"errorMessage":"Filter string exceeds the maximum allowed length.","messagePattern":"Filter string exceeds the maximum allowed length\\.","errorType":"exception","errorClass":"VectorStoreOperationException","httpStatus":null,"severity":"error","filePath":"python/semantic_kernel/connectors/in_memory.py","lineNumber":751,"sourceCode":"            ]\n        except Exception as e:\n            raise VectorStoreOperationException(f\"Error evaluating filter: {e}\") from e\n        filtered_records: dict[TKey, AttributeDict] = {}\n        for key, record in self.inner_storage.items():\n            for filter in callable_filters:\n                if self._run_filter(filter, record):\n                    filtered_records[key] = record\n        return filtered_records\n\n    def _parse_and_validate_filter(self, filter_str: str) -> Callable:\n        \"\"\"Parse and validate a string filter as a lambda expression, then return the callable.\n\n        Uses an allowlist approach - only explicitly permitted AST node types and function names\n        are allowed. This can be customized by overriding `allowed_filter_ast_nodes` and\n        `allowed_filter_functions` class attributes.\n        \"\"\"\n        if len(filter_str) > self.max_filter_source_length:\n            raise VectorStoreOperationException(\"Filter string exceeds the maximum allowed length.\")\n\n        try:\n            tree = ast.parse(filter_str, mode=\"eval\")\n        except SyntaxError as e:\n            raise VectorStoreOperationException(f\"Filter string is not valid Python: {e}\") from e\n\n        # Only allow lambda expressions at the top level\n        if not (isinstance(tree, ast.Expression) and isinstance(tree.body, ast.Lambda)):\n            raise VectorStoreOperationException(\n                \"Filter string must be a lambda expression, e.g. 'lambda x: x.key == 1'\"\n            )\n\n        # Get the lambda parameter name(s) to allow them as valid Name nodes\n        lambda_node = tree.body\n        lambda_param_names = {arg.arg for arg in lambda_node.args.args}\n        lambda_param_order = [arg.arg for arg in lambda_node.args.args]\n        # Walk the AST to validate all nodes against the allowlist\n        for node_count, node in enumerate(ast.walk(tree), start=1):","sourceCodeStart":733,"sourceCodeEnd":769,"githubUrl":"https://github.com/microsoft/semantic-kernel/blob/c028a0c7dc4f0814cdcbaba9d998f187a41197bf/python/semantic_kernel/connectors/in_memory.py#L733-L769","documentation":"Thrown by _parse_and_validate_filter (in_memory.py:750-751) when the raw filter string is longer than max_filter_source_length (default 2048 characters), before any parsing. It is the first guard against oversized filter input.","triggerScenarios":"Passing a very long lambda string, typically one containing a huge inline literal collection or very long string constants.","commonSituations":"Inlining a large ID allowlist into the filter; generating filters from templates that balloon in size; lowering the limit on a shared collection.","solutions":["Shorten the filter string (e.g. move large literal sets out of the filter).","Pass options.filter as a Python callable instead of a string to bypass the source-length limit for trusted logic.","Raise max_filter_source_length on the collection instance if a genuinely long filter is required.","Split a single long filter into a list of shorter filters (options.filter accepts a list, OR-semantics)."],"exampleFix":"# before\ncollection.max_filter_source_length = 2048  # default\nopts = VectorSearchOptions(filter=f\"lambda x: x.id in {str(list(range(5000)))}\")  # huge\n# after\nallowed = set(range(5000))\nopts = VectorSearchOptions(filter=lambda x: x.id in allowed)  # callable, no length limit","handlingStrategy":"validation","validationCode":"def within_source_limit(expr: str, cap: int = 2048) -> bool:\n    return len(expr) <= cap","typeGuard":null,"tryCatchPattern":"try:\n    results = await collection.search(search_type=SearchType.VECTOR, options=opts)\nexcept VectorStoreOperationException as e:\n    logger.warning(\"filter rejected: %s\", e.__cause__ or e)\n    results = None","preventionTips":["Never inline huge data sets into filter strings.","Use callable filters for large, trusted logic.","Tune max_filter_source_length deliberately, not as a reflex."],"tags":["filter","in-memory","resource-limits","semantic-kernel"],"backgroundTag":null,"analyzedSha":"c028a0c7dc4f0814cdcbaba9d998f187a41197bf","analyzedAt":"2026-08-13T13:48:05.040Z","schemaVersion":2},"datasetVersion":"2026-08-13T14:17:21.547Z"}