zylon-ai/private-gpt · error · ValueError

Remote tokenizer response did not contain a valid 'tokens' f

Error message

Remote tokenizer response did not contain a valid 'tokens' field

What it means

Raised by the static helper RemoteTokenizeTokenizer._extract_tokens while parsing a tokenize response. The contract is strict: the JSON payload must be a dict containing key 'tokens' whose value is a list where every element is an int. Anything else — missing key, wrong key name, list of strings/floats/nulls, or a non-dict payload — raises this ValueError. It almost always indicates a contract mismatch with the remote tokenizer HTTP service, not a transient network problem.

Source

Thrown at private_gpt/components/llm/tokenizers/remote.py:342

    def _build_url(self, path: str) -> str:
        return f"{self.api_base}/{path.lstrip('/')}"

    def _headers(self) -> dict[str, str]:
        headers = {"Content-Type": "application/json"}
        if self.api_key:
            headers["Authorization"] = f"Bearer {self.api_key}"
        return headers

    @staticmethod
    def _extract_tokens(payload: Any) -> list[int]:
        if isinstance(payload, dict):
            tokens = payload.get("tokens")
            if isinstance(tokens, list) and all(
                isinstance(token, int) for token in tokens
            ):
                return tokens
        raise ValueError(
            "Remote tokenizer response did not contain a valid 'tokens' field"
        )

    @staticmethod
    def _extract_text(payload: Any) -> str:
        if isinstance(payload, dict):
            for key in ("content", "text"):
                value = payload.get(key)
                if isinstance(value, str):
                    return value
        raise ValueError(
            "Remote detokenize response did not contain a supported text field"
        )

View on GitHub (pinned to 4a030776a3)

Solutions

1. Verify the remote endpoint actually implements the expected contract: JSON object with 'tokens': list[int].
2. Curl the endpoint manually and inspect the exact response shape; fix the server or the URL accordingly.
3. If tokens come back as strings/floats, normalize them server-side to integers.
4. Ensure error responses use non-2xx status codes so they surface as HTTP errors instead of reaching _extract_tokens.

Example fix

# before (server returns)
{"token_ids": [1, 2, 3]}  # ValueError: no 'tokens' field

# after (server returns)
{"tokens": [1, 2, 3]}

def is_valid_tokens_payload(payload: Any) -> bool:
    return (isinstance(payload, dict)
            and isinstance(payload.get('tokens'), list)
            and all(isinstance(t, int) for t in payload['tokens']))

try:
    ids = tok.tokenize(texts=text).input_ids
except ValueError as e:
    if 'tokens' in str(e):
        log_raw_response(); raise  # contract mismatch with tokenizer service

When it happens

Trigger: POSTing text to the remote tokenize endpoint and receiving JSON without a 'tokens' key (e.g. {'token_ids': [...]} or {'count': 42}); receiving tokens as strings ('123') or floats; the endpoint returning an error object {'error': ...} with HTTP 200; the URL pointing at a different API (e.g. detokenize or an OpenAI-compatible endpoint).

Common situations: Pointing the remote tokenizer URL at a custom/in-house service with a different schema; version drift between the tokenizer server and this client; a proxy or gateway rewriting the response body; misconfigured URL hitting an unrelated route.

Related errors

• Remote detokenize response did not contain a supported text
• INVALID_REQUEST_ERROR
• REQUEST_TOO_LARGE_ERROR
• Last item is a FlexibleModel, expected a specific output_cls
• Tokenizer is required and must support apply_chat_template:

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/e23c38b420930693. Report an issue: GitHub.