{"record":{"id":"53ea92e2c9deb057","repo":"oobabooga/textgen","slug":"error-failed-to-load-embedding-model-model","errorCode":null,"errorMessage":"Error: Failed to load embedding model: {model}","messagePattern":"Error: Failed to load embedding model: (.+?)","errorType":"exception","errorClass":"ServiceUnavailableError","httpStatus":503,"severity":"error","filePath":"modules/api/embeddings.py","lineNumber":53,"sourceCode":"        from sentence_transformers import SentenceTransformer\n    except ModuleNotFoundError:\n        logger.error(\"The sentence_transformers module has not been found. Please install it manually with pip install -U sentence-transformers.\")\n        raise ModuleNotFoundError\n\n    initialize_embedding_params()\n    global embeddings_device, embeddings_model\n    try:\n        logger.info(f\"Try embedding model: {model} on {embeddings_device}\")\n        if 'jina-embeddings' in model:\n            embeddings_model = AutoModel.from_pretrained(model, trust_remote_code=shared.args.trust_remote_code)\n            embeddings_model = embeddings_model.to(embeddings_device)\n        else:\n            embeddings_model = SentenceTransformer(model, device=embeddings_device)\n\n        logger.info(f\"Loaded embedding model: {model}\")\n    except Exception as e:\n        embeddings_model = None\n        raise ServiceUnavailableError(f\"Error: Failed to load embedding model: {model}\", internal_message=repr(e))\n\n\ndef get_embeddings_model():\n    initialize_embedding_params()\n    global embeddings_model, st_model\n    if st_model and not embeddings_model:\n        load_embedding_model(st_model)  # lazy load the model\n\n    return embeddings_model\n\n\ndef get_embeddings_model_name() -> str:\n    initialize_embedding_params()\n    global st_model\n    return st_model\n\n\ndef get_embeddings(input: list) -> np.ndarray:","sourceCodeStart":35,"sourceCodeEnd":71,"githubUrl":"https://github.com/oobabooga/textgen/blob/ed888c71f221df552750e1834b3654abab8ae345/modules/api/embeddings.py#L35-L71","documentation":"load_embedding_model tries to load the configured embedding model with SentenceTransformer (or AutoModel with trust_remote_code for jina-embeddings) onto the configured embeddings device. Any exception during load is re-raised as ServiceUnavailableError (HTTP 503) wrapping the original exception in internal_message; the global embeddings_model is reset to None so subsequent calls retry.","triggerScenarios":"POST /v1/embeddings when the model configured (env HF_TOKEN gated model, non-existent repo id, or a local path that is absent) fails to download/load; CUDA device mismatch (embeddings_device=cuda with no GPU); a jina model loaded without --trust-remote-code; offline machine with no cached snapshot.","commonSituations":"Wrong or misspelled model name in the --api embedding model setting; network/proxy blocked HF downloads; no HF token for gated models (e.g. some sentence-transformers variants); sentence_transformers/transformers version conflict; disk-full during model download.","solutions":["Check server logs for internal_message=repr(e) to see the underlying cause (HTTP 401/403, 404, CUDA error, ImportError).","Set a valid, public embedding model id (e.g. sentence-transformers/all-MiniLM-L6-v2) in the API embedding model setting.","For jina models, launch the server with --trust-remote-code.","Ensure the device exists: set embeddings device to cpu if no GPU, or fix CUDA install.","Pre-download the model (huggingface-cli download <model>) or point to a local snapshot path when offline."],"exampleFix":"# before: server started with missing/gated model, embeddings calls 503\npython server.py --api --embeddings-model somemodel/all-MiniLM\n\n# after: valid public model on cpu\npython server.py --api --embeddings-model sentence-transformers/all-MiniLM-L6-v2 --embeddings-device cpu","handlingStrategy":"retry","validationCode":"def embeddings_ready(base_url: str, model: str) -> bool:\n    import requests\n    r = requests.get(f'{base_url}/v1/internal/model/info', timeout=10)\n    return r.ok and any(model in m.get('id', '') for m in r.json().get('data', []))","typeGuard":null,"tryCatchPattern":"try:\n    emb = client.embeddings.create(model=model, input=text)\nexcept openai.APIStatusError as e:\n    if e.status_code == 503 and 'Failed to load embedding model' in str(e):\n        # check server logs for internal_message; fix config, then retry once after fixing\n        raise RuntimeError(f'Embedding model {model} failed to load; see server log')\n    raise","preventionTips":["Pre-download the embedding model and point the server at a local path for air-gapped setups.","Set --embeddings-device cpu when no GPU is guaranteed.","Pass --trust-remote-code only for models that need it (jina).","Check server logs for internal_message to see the wrapped root cause before retrying."],"tags":["embeddings","model-loading","huggingface","service-unavailable"],"backgroundTag":null,"analyzedSha":"ed888c71f221df552750e1834b3654abab8ae345","analyzedAt":"2026-08-15T05:24:21.000Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}