{"record":{"id":"b90bef3f4edbbd29","repo":"zylon-ai/private-gpt","slug":"model-server-is-not-available-or-request-failed-b90bef","errorCode":null,"errorMessage":"Model server is not available or request failed.","messagePattern":"Model server is not available or request failed\\.","errorType":"exception","errorClass":"ModelNotAvailableError","httpStatus":null,"severity":"critical","filePath":"private_gpt/components/multimodality/image_handler.py","lineNumber":405,"sourceCode":"                            f\"Request too large on attempt {count}, will retry with reduced quality\"\n                        )\n                        if count >= self._num_max_retries:\n                            return e\n                        raise\n\n                async def _call_with_semaphore() -> Any:\n                    if semaphore_manager:\n                        return await semaphore_manager.execute(\n                            task_func=_call, priority=0\n                        )\n                    return await _call()\n\n                result = await retry(_call_with_semaphore)\n                if isinstance(result, Exception):\n                    raise result\n                return result\n        except MODEL_NOT_AVAILABLE_EXCEPTION_TYPES as e:\n            raise ModelNotAvailableError(\n                \"Model server is not available or request failed.\"\n            ) from e\n\n    async def _infer_strategy(\n        self, image_blocks: list[ImageBlock], **kwargs: Any\n    ) -> ExtractionStrategy:\n        strategy_prompt = self._prompt_builder.create_image_strategy_prompt()\n\n        messages = [\n            ChatMessage(\n                role=MessageRole.SYSTEM,\n                blocks=[TextBlock(text=strategy_prompt.format())],\n            ),\n            ChatMessage(\n                role=MessageRole.USER,\n                blocks=[\n                    TextBlock(text=\"Analyze the following image:\"),\n                    *image_blocks,","sourceCodeStart":387,"sourceCodeEnd":423,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/multimodality/image_handler.py#L387-L423","documentation":"The image handler wraps its structured-chat call in tenacity retries plus an optional `SemaphoreManager` for concurrency limiting, and catches `MODEL_NOT_AVAILABLE_EXCEPTION_TYPES` (`ConnectionError`, `TimeoutError`, `OSError`, grpc `AioRpcError`, Triton `InferenceServerException`). When all retries fail with one of these, it re-raises as `ModelNotAvailableError` from the original cause.","triggerScenarios":"Image extraction request against a down/unreachable model server; inference endpoint timing out under load (large image payloads) until retries exhaust; gRPC/Triton backend returning transport errors; semaphore-throttled requests still failing after the retry budget.","commonSituations":"vLLM/Triton server OOM-killed by large vision models; slow image inference exceeding client timeout on every attempt; misconfigured endpoint URL; network partition between app and inference service; concurrent load exceeding server capacity.","solutions":["Check model server health and restart it if crashed (vision models often OOM)","Increase request timeout and/or retry attempts/backoff for image workloads, which are slower than text","Verify base_url/port and network reachability of the multimodal inference endpoint","Reduce concurrency (lower semaphore limits) so the server is not overloaded","Handle `ModelNotAvailableError` upstream and surface a retryable 503"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    result = await image_handler.extract(...)\nexcept ModelNotAvailableError:\n    logger.warning(\"vision backend unavailable; queueing for retry\")\n    await backoff_and_reenqueue(job)","preventionTips":["Raise client timeouts for image inference; vision calls are much slower than text","Cap concurrency below the inference server's capacity to avoid timeout cascades","Watch memory on the model server (vision models OOM) and alert before crashes"],"tags":["network","llm","availability","vision","retry-exhausted"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}