{"record":{"id":"e23c38b420930693","repo":"zylon-ai/private-gpt","slug":"remote-tokenizer-response-did-not-contain-a-valid","errorCode":null,"errorMessage":"Remote tokenizer response did not contain a valid 'tokens' field","messagePattern":"Remote tokenizer response did not contain a valid 'tokens' field","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/remote.py","lineNumber":342,"sourceCode":"\n    def _build_url(self, path: str) -> str:\n        return f\"{self.api_base}/{path.lstrip('/')}\"\n\n    def _headers(self) -> dict[str, str]:\n        headers = {\"Content-Type\": \"application/json\"}\n        if self.api_key:\n            headers[\"Authorization\"] = f\"Bearer {self.api_key}\"\n        return headers\n\n    @staticmethod\n    def _extract_tokens(payload: Any) -> list[int]:\n        if isinstance(payload, dict):\n            tokens = payload.get(\"tokens\")\n            if isinstance(tokens, list) and all(\n                isinstance(token, int) for token in tokens\n            ):\n                return tokens\n        raise ValueError(\n            \"Remote tokenizer response did not contain a valid 'tokens' field\"\n        )\n\n    @staticmethod\n    def _extract_text(payload: Any) -> str:\n        if isinstance(payload, dict):\n            for key in (\"content\", \"text\"):\n                value = payload.get(key)\n                if isinstance(value, str):\n                    return value\n        raise ValueError(\n            \"Remote detokenize response did not contain a supported text field\"\n        )\n","sourceCodeStart":324,"sourceCodeEnd":356,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/remote.py#L324-L356","documentation":"Raised by the static helper RemoteTokenizeTokenizer._extract_tokens while parsing a tokenize response. The contract is strict: the JSON payload must be a dict containing key 'tokens' whose value is a list where every element is an int. Anything else — missing key, wrong key name, list of strings/floats/nulls, or a non-dict payload — raises this ValueError. It almost always indicates a contract mismatch with the remote tokenizer HTTP service, not a transient network problem.","triggerScenarios":"POSTing text to the remote tokenize endpoint and receiving JSON without a 'tokens' key (e.g. {'token_ids': [...]} or {'count': 42}); receiving tokens as strings ('123') or floats; the endpoint returning an error object {'error': ...} with HTTP 200; the URL pointing at a different API (e.g. detokenize or an OpenAI-compatible endpoint).","commonSituations":"Pointing the remote tokenizer URL at a custom/in-house service with a different schema; version drift between the tokenizer server and this client; a proxy or gateway rewriting the response body; misconfigured URL hitting an unrelated route.","solutions":["Verify the remote endpoint actually implements the expected contract: JSON object with 'tokens': list[int].","Curl the endpoint manually and inspect the exact response shape; fix the server or the URL accordingly.","If tokens come back as strings/floats, normalize them server-side to integers.","Ensure error responses use non-2xx status codes so they surface as HTTP errors instead of reaching _extract_tokens."],"exampleFix":"# before (server returns)\n{\"token_ids\": [1, 2, 3]}  # ValueError: no 'tokens' field\n\n# after (server returns)\n{\"tokens\": [1, 2, 3]}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":"def is_valid_tokens_payload(payload: Any) -> bool:\n    return (isinstance(payload, dict)\n            and isinstance(payload.get('tokens'), list)\n            and all(isinstance(t, int) for t in payload['tokens']))","tryCatchPattern":"try:\n    ids = tok.tokenize(texts=text).input_ids\nexcept ValueError as e:\n    if 'tokens' in str(e):\n        log_raw_response(); raise  # contract mismatch with tokenizer service","preventionTips":["Contract-test the tokenizer service (assert 'tokens': list[int]) in CI alongside the client.","Make the tokenizer server return non-2xx on errors so schema bugs surface as HTTP errors, not parse errors."],"tags":["network","api-contract","tokenizer","json"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}