zylon-ai/private-gpt · error · ValueError

Last item is a FlexibleModel, expected a specific…

Error message

Last item is a FlexibleModel, expected a specific output_cls.

What it means

ValueError raised by StructuredChatMixin.structured_predict: it consumes the full stream_structured_predict output and, if the last parsed item is a FlexibleModel (the lenient fallback model used when the streamed JSON could not validate against output_cls), it refuses to return it. This signals the model's final output never conformed to the requested Pydantic schema.

Solutions

Inspect the raw completion (log stream chunks) to see what JSON the model actually produced versus output_cls.
Simplify the output schema: fewer required fields, more permissive types, clear field descriptions.
Strengthen the prompt to demand raw JSON only (no prose/markdown fences).
Switch to a model with native structured-output support, or use structured_chat with allow_flexible=True and validate/repair manually.

Example fix

# before
class Answer(BaseModel):
    confidence: float  # model keeps omitting

result = llm.structured_predict(Answer, prompt)

# after
class Answer(BaseModel):
    confidence: float | None = None

result = llm.structured_predict(Answer, prompt)

Defensive patterns

Strategy: fallback

Try / catch

try:
    result = llm.structured_predict(Answer, prompt)
except ValueError as e:
    if 'FlexibleModel' in str(e):
        raw = llm.structured_chat(Answer, messages, allow_flexible=True)
        result = repair_and_validate(raw, Answer)  # coerce fields, retry once

Prevention

Design schemas the target model can actually satisfy (optional fields, lenient types).
Keep a repair/retry path: capture the flexible output, fix it up, validate against output_cls programmatically.

When it happens

Trigger: Calling structured_predict where the LLM's accumulated JSON fails output_cls validation at every stream step, so only FlexibleModel instances are emitted — malformed JSON, wrong field names/types, or the model adding prose around the JSON.

Common situations: Weak models ignoring schema instructions; schemas with strict constraints (required fields, enums, formats) the model violates; responses wrapped in markdown fences or text that breaks parsing; truncated streams cutting required fields.

Related errors

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/678127f5b9b0c0d0. Report an issue: GitHub.

Appendix: source

Thrown at private_gpt/components/llm/custom/structured_mixin.py:76

    def structured_predict(
        self,
        output_cls: type[Model],
        prompt: PromptTemplate,
        llm_kwargs: dict[str, Any] | None = None,
        **prompt_args: Any,
    ) -> Model:
        items: list[Model | FlexibleModel] = list(
            self.stream_structured_predict(
                output_cls=output_cls,
                prompt=prompt,
                llm_kwargs=llm_kwargs,
                **prompt_args,
            )
        )
        last_item: Model | FlexibleModel = items[-1]
        if isinstance(last_item, FlexibleModel):
            raise ValueError(
                "Last item is a FlexibleModel, expected a specific output_cls."
            )
        return last_item  # type: ignore[return-value]

    def stream_structured_predict(
        self,
        output_cls: type[Model],
        prompt: PromptTemplate,
        llm_kwargs: dict[str, Any] | None = None,
        **prompt_args: Any,
    ) -> Generator[Model | FlexibleModel, None, None]:
        messages = [
            ChatMessage(
                role=MessageRole.USER,
                content=prompt.format(**prompt_args),
            )
        ]
        return self.stream_structured_chat(

View on GitHub (pinned to 4a030776a3)