zylon-ai/private-gpt · error · ValueError

Remote detokenize response did not contain a supported text

Error message

Remote detokenize response did not contain a supported text field

What it means

Raised by RemoteTokenizeTokenizer._extract_text while parsing a detokenize response. The accepted contract is a dict with either a 'content' or 'text' key holding a string (checked in that order). If both are absent, hold non-string values, or the payload is not a dict, this ValueError fires. Like error 145 it signals an API contract mismatch with the remote tokenizer service on the detokenize direction.

Source

Thrown at private_gpt/components/llm/tokenizers/remote.py:353

    def _extract_tokens(payload: Any) -> list[int]:
        if isinstance(payload, dict):
            tokens = payload.get("tokens")
            if isinstance(tokens, list) and all(
                isinstance(token, int) for token in tokens
            ):
                return tokens
        raise ValueError(
            "Remote tokenizer response did not contain a valid 'tokens' field"
        )

    @staticmethod
    def _extract_text(payload: Any) -> str:
        if isinstance(payload, dict):
            for key in ("content", "text"):
                value = payload.get(key)
                if isinstance(value, str):
                    return value
        raise ValueError(
            "Remote detokenize response did not contain a supported text field"
        )

View on GitHub (pinned to 4a030776a3)

Solutions

Make the detokenize endpoint return {"content": "..."} or {"text": "..."} with a string value.
Inspect the raw response with curl to find the actual field name and align server/client.
Return proper HTTP error statuses from the server so failures do not masquerade as malformed success payloads.
If the field is nested (e.g. {'result': {'text': ...}}), unwrap it server-side or at the proxy.

Example fix

# before (server returns)
{"decoded": "hello world"}  # ValueError: no supported text field

# after (server returns)
{"content": "hello world"}

Defensive patterns

Strategy: try-catch

Type guard

def is_valid_text_payload(payload: Any) -> bool:
    return isinstance(payload, dict) and isinstance(
        payload.get('content', payload.get('text')), str)

Try / catch

try:
    text = await tok.adetokenize(ids)
except ValueError as e:
    if 'text field' in str(e):
        log_payload(); raise

Prevention

Pin the tokenizer service version next to the client so response-schema drift is caught in upgrades.
Return {'content': str} from detokenize and cover it with an integration test.

When it happens

Trigger: Calling the remote detokenize path and receiving JSON without 'content'/'text' (e.g. {'decoded': "..."}), with a non-string value (null, number), or a top-level list/string instead of an object; endpoint returning an error body with HTTP 200.

Common situations: Custom tokenizer server using a different field name; server upgrade changing the response schema; URL pointing to a tokenize-only endpoint; gateways injecting wrapper objects around the real payload.

Related errors

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/1da40bb57bf09a20. Report an issue: GitHub.