zylon-ai/private-gpt · critical · ModelNotAvailableError

Model server is not available or request failed.

Error message

Model server is not available or request failed.

What it means

The image handler wraps its structured-chat call in tenacity retries plus an optional `SemaphoreManager` for concurrency limiting, and catches `MODEL_NOT_AVAILABLE_EXCEPTION_TYPES` (`ConnectionError`, `TimeoutError`, `OSError`, grpc `AioRpcError`, Triton `InferenceServerException`). When all retries fail with one of these, it re-raises as `ModelNotAvailableError` from the original cause.

Source

Thrown at private_gpt/components/multimodality/image_handler.py:405

                            f"Request too large on attempt {count}, will retry with reduced quality"
                        )
                        if count >= self._num_max_retries:
                            return e
                        raise

                async def _call_with_semaphore() -> Any:
                    if semaphore_manager:
                        return await semaphore_manager.execute(
                            task_func=_call, priority=0
                        )
                    return await _call()

                result = await retry(_call_with_semaphore)
                if isinstance(result, Exception):
                    raise result
                return result
        except MODEL_NOT_AVAILABLE_EXCEPTION_TYPES as e:
            raise ModelNotAvailableError(
                "Model server is not available or request failed."
            ) from e

    async def _infer_strategy(
        self, image_blocks: list[ImageBlock], **kwargs: Any
    ) -> ExtractionStrategy:
        strategy_prompt = self._prompt_builder.create_image_strategy_prompt()

        messages = [
            ChatMessage(
                role=MessageRole.SYSTEM,
                blocks=[TextBlock(text=strategy_prompt.format())],
            ),
            ChatMessage(
                role=MessageRole.USER,
                blocks=[
                    TextBlock(text="Analyze the following image:"),
                    *image_blocks,

View on GitHub (pinned to 4a030776a3)

Solutions

1. Check model server health and restart it if crashed (vision models often OOM)
2. Increase request timeout and/or retry attempts/backoff for image workloads, which are slower than text
3. Verify base_url/port and network reachability of the multimodal inference endpoint
4. Reduce concurrency (lower semaphore limits) so the server is not overloaded
5. Handle `ModelNotAvailableError` upstream and surface a retryable 503

try:
    result = await image_handler.extract(...)
except ModelNotAvailableError:
    logger.warning("vision backend unavailable; queueing for retry")
    await backoff_and_reenqueue(job)

When it happens

Trigger: Image extraction request against a down/unreachable model server; inference endpoint timing out under load (large image payloads) until retries exhaust; gRPC/Triton backend returning transport errors; semaphore-throttled requests still failing after the retry budget.

Common situations: vLLM/Triton server OOM-killed by large vision models; slow image inference exceeding client timeout on every attempt; misconfigured endpoint URL; network partition between app and inference service; concurrent load exceeding server capacity.

Related errors

• Model server is not available or request failed.
• Failed to describe images in the message.
• LLM does not support structured chat.
• OVERLOADED_CONDENSATION_ERROR
• Audio blocks found but no audio-capable LLM provided.

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/b90bef3f4edbbd29. Report an issue: GitHub.