zylon-ai/private-gpt · error · ValueError

tokenizer_fn must be provided.

Error message

tokenizer_fn must be provided.

What it means

Raised by the same TrimmingMemory model_validator when tokenizer_fn is None. The trimming strategy needs to count tokens per message to decide what fits under token_limit, and no default tokenizer is wired yet (the TODO comment confirms a default is planned but absent), so a missing tokenizer_fn is a hard construction error rather than a fallback.

Source

Thrown at private_gpt/components/memory/trimming_memory.py:86

    @classmethod
    def class_name(cls) -> str:
        """Get class name."""
        return "TrimmingMemory"

    @model_validator(mode="before")
    @classmethod
    def validate_memory(cls, values: dict[str, Any]) -> dict[str, Any]:
        """Validate memory configuration."""
        # Validate token limit
        token_limit = values.get("token_limit", -1)
        if token_limit < 1:
            raise ValueError("Token limit must be set and greater than 0.")

        # Validate tokenizer
        tokenizer_fn = values.get("tokenizer_fn")
        if tokenizer_fn is None:
            # TODO: Replace with a default tokenizer function
            raise ValueError("tokenizer_fn must be provided.")

        # Validate text splitter
        text_splitter = values.get("text_splitter")
        if text_splitter is None:
            values["text_splitter"] = _default_text_splitter

        # Validate strategy-specific constraints
        trim_strategy = values.get("trim_strategy", TrimStrategy.LAST)
        start_on = values.get("start_on")
        include_system = values.get("include_system", True)

        if start_on and trim_strategy == TrimStrategy.FIRST:
            raise ValueError("start_on can only be used with 'last' strategy")

        if include_system and trim_strategy == TrimStrategy.FIRST:
            raise ValueError("include_system can only be used with 'last' strategy")

        return values

View on GitHub (pinned to 4a030776a3)

Solutions

1. Provide a tokenizer_fn, e.g. a callable from your tokenizer: tokenizer_fn=lambda text: len(tokenizer.tokenize(texts=text).input_ids) or str.split for a cheap approximation.
2. Prefer TrimmingMemory.from_defaults / Memory.from_defaults, which wire the tokenizer function for you.
3. If a config field feeds tokenizer_fn, make it required at config-load time.

Example fix

# before
memory = TrimmingMemory(token_limit=2048)

# after
memory = TrimmingMemory(
    token_limit=2048,
    tokenizer_fn=lambda text: len(tok.tokenize(texts=text).input_ids),
)

if tokenizer_fn is None:
    tokenizer_fn = lambda text: len(str(text).split())  # word-count fallback
mem = TrimmingMemory(token_limit=2048, tokenizer_fn=tokenizer_fn)

try:
    mem = TrimmingMemory(token_limit=2048)
except ValidationError as e:
    if 'tokenizer_fn' in str(e):
        mem = TrimmingMemory(token_limit=2048, tokenizer_fn=default_tokenizer_fn)

When it happens

Trigger: Constructing TrimmingMemory(...) directly without tokenizer_fn; passing tokenizer_fn=None explicitly; copying constructor examples that predate the tokenizer_fn requirement.

Common situations: Upgrading to a private-gpt version that added the tokenizer_fn requirement without updating call sites; tests constructing the memory with only token_limit; integrations relying on a former implicit default tokenizer.

Related errors

• Token limit must be set and greater than 0.
• 'oneOf' must be an array of schemas
• 'anyOf' must be an array of schemas
• 'allOf' must be an array of schemas
• Schema must define a 'type' field

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/2ac80133b9f4abaa. Report an issue: GitHub.