zylon-ai/private-gpt · error · ValueError

Last item is a FlexibleModel, expected a specific output_cls

Error message

Last item is a FlexibleModel, expected a specific output_cls.

What it means

ValueError raised by StructuredChatMixin.structured_predict: it consumes the full stream_structured_predict output and, if the last parsed item is a FlexibleModel (the lenient fallback model used when the streamed JSON could not validate against output_cls), it refuses to return it. This signals the model's final output never conformed to the requested Pydantic schema.

Source

Thrown at private_gpt/components/llm/custom/structured_mixin.py:76

    def structured_predict(
        self,
        output_cls: type[Model],
        prompt: PromptTemplate,
        llm_kwargs: dict[str, Any] | None = None,
        **prompt_args: Any,
    ) -> Model:
        items: list[Model | FlexibleModel] = list(
            self.stream_structured_predict(
                output_cls=output_cls,
                prompt=prompt,
                llm_kwargs=llm_kwargs,
                **prompt_args,
            )
        )
        last_item: Model | FlexibleModel = items[-1]
        if isinstance(last_item, FlexibleModel):
            raise ValueError(
                "Last item is a FlexibleModel, expected a specific output_cls."
            )
        return last_item  # type: ignore[return-value]

    def stream_structured_predict(
        self,
        output_cls: type[Model],
        prompt: PromptTemplate,
        llm_kwargs: dict[str, Any] | None = None,
        **prompt_args: Any,
    ) -> Generator[Model | FlexibleModel, None, None]:
        messages = [
            ChatMessage(
                role=MessageRole.USER,
                content=prompt.format(**prompt_args),
            )
        ]
        return self.stream_structured_chat(

View on GitHub (pinned to 4a030776a3)

Solutions

1. Inspect the raw completion (log stream chunks) to see what JSON the model actually produced versus output_cls.
2. Simplify the output schema: fewer required fields, more permissive types, clear field descriptions.
3. Strengthen the prompt to demand raw JSON only (no prose/markdown fences).
4. Switch to a model with native structured-output support, or use structured_chat with allow_flexible=True and validate/repair manually.

Example fix

# before
class Answer(BaseModel):
    confidence: float  # model keeps omitting

result = llm.structured_predict(Answer, prompt)

# after
class Answer(BaseModel):
    confidence: float | None = None

result = llm.structured_predict(Answer, prompt)

try:
    result = llm.structured_predict(Answer, prompt)
except ValueError as e:
    if 'FlexibleModel' in str(e):
        raw = llm.structured_chat(Answer, messages, allow_flexible=True)
        result = repair_and_validate(raw, Answer)  # coerce fields, retry once

When it happens

Trigger: Calling structured_predict where the LLM's accumulated JSON fails output_cls validation at every stream step, so only FlexibleModel instances are emitted — malformed JSON, wrong field names/types, or the model adding prose around the JSON.

Common situations: Weak models ignoring schema instructions; schemas with strict constraints (required fields, enums, formats) the model violates; responses wrapped in markdown fences or text that breaks parsing; truncated streams cutting required fields.

Related errors

• INVALID_REQUEST_ERROR
• REQUEST_TOO_LARGE_ERROR
• Schema must define a 'type' field
• Array schemas must define 'items'
• Array 'items' must be a dictionary representing JSON Schema

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/678127f5b9b0c0d0. Report an issue: GitHub.