{"record":{"id":"2ac80133b9f4abaa","repo":"zylon-ai/private-gpt","slug":"tokenizer-fn-must-be-provided","errorCode":null,"errorMessage":"tokenizer_fn must be provided.","messagePattern":"tokenizer_fn must be provided\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/memory/trimming_memory.py","lineNumber":86,"sourceCode":"    @classmethod\n    def class_name(cls) -> str:\n        \"\"\"Get class name.\"\"\"\n        return \"TrimmingMemory\"\n\n    @model_validator(mode=\"before\")\n    @classmethod\n    def validate_memory(cls, values: dict[str, Any]) -> dict[str, Any]:\n        \"\"\"Validate memory configuration.\"\"\"\n        # Validate token limit\n        token_limit = values.get(\"token_limit\", -1)\n        if token_limit < 1:\n            raise ValueError(\"Token limit must be set and greater than 0.\")\n\n        # Validate tokenizer\n        tokenizer_fn = values.get(\"tokenizer_fn\")\n        if tokenizer_fn is None:\n            # TODO: Replace with a default tokenizer function\n            raise ValueError(\"tokenizer_fn must be provided.\")\n\n        # Validate text splitter\n        text_splitter = values.get(\"text_splitter\")\n        if text_splitter is None:\n            values[\"text_splitter\"] = _default_text_splitter\n\n        # Validate strategy-specific constraints\n        trim_strategy = values.get(\"trim_strategy\", TrimStrategy.LAST)\n        start_on = values.get(\"start_on\")\n        include_system = values.get(\"include_system\", True)\n\n        if start_on and trim_strategy == TrimStrategy.FIRST:\n            raise ValueError(\"start_on can only be used with 'last' strategy\")\n\n        if include_system and trim_strategy == TrimStrategy.FIRST:\n            raise ValueError(\"include_system can only be used with 'last' strategy\")\n\n        return values","sourceCodeStart":68,"sourceCodeEnd":104,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/memory/trimming_memory.py#L68-L104","documentation":"Raised by the same TrimmingMemory model_validator when tokenizer_fn is None. The trimming strategy needs to count tokens per message to decide what fits under token_limit, and no default tokenizer is wired yet (the TODO comment confirms a default is planned but absent), so a missing tokenizer_fn is a hard construction error rather than a fallback.","triggerScenarios":"Constructing TrimmingMemory(...) directly without tokenizer_fn; passing tokenizer_fn=None explicitly; copying constructor examples that predate the tokenizer_fn requirement.","commonSituations":"Upgrading to a private-gpt version that added the tokenizer_fn requirement without updating call sites; tests constructing the memory with only token_limit; integrations relying on a former implicit default tokenizer.","solutions":["Provide a tokenizer_fn, e.g. a callable from your tokenizer: tokenizer_fn=lambda text: len(tokenizer.tokenize(texts=text).input_ids) or str.split for a cheap approximation.","Prefer TrimmingMemory.from_defaults / Memory.from_defaults, which wire the tokenizer function for you.","If a config field feeds tokenizer_fn, make it required at config-load time."],"exampleFix":"# before\nmemory = TrimmingMemory(token_limit=2048)\n\n# after\nmemory = TrimmingMemory(\n    token_limit=2048,\n    tokenizer_fn=lambda text: len(tok.tokenize(texts=text).input_ids),\n)","handlingStrategy":"validation","validationCode":"if tokenizer_fn is None:\n    tokenizer_fn = lambda text: len(str(text).split())  # word-count fallback\nmem = TrimmingMemory(token_limit=2048, tokenizer_fn=tokenizer_fn)","typeGuard":null,"tryCatchPattern":"try:\n    mem = TrimmingMemory(token_limit=2048)\nexcept ValidationError as e:\n    if 'tokenizer_fn' in str(e):\n        mem = TrimmingMemory(token_limit=2048, tokenizer_fn=default_tokenizer_fn)","preventionTips":["Centralize one tokenizer_fn and pass it to every memory you construct.","Construct memory via from_defaults so the tokenizer wiring is done for you."],"tags":["memory","validation","tokenizer","pydantic"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}