oobabooga/textgen · error · ServiceUnavailableError

Error: Failed to load embedding model: {model}

Error message

Error: Failed to load embedding model: {model}

What it means

load_embedding_model tries to load the configured embedding model with SentenceTransformer (or AutoModel with trust_remote_code for jina-embeddings) onto the configured embeddings device. Any exception during load is re-raised as ServiceUnavailableError (HTTP 503) wrapping the original exception in internal_message; the global embeddings_model is reset to None so subsequent calls retry.

Source

Thrown at modules/api/embeddings.py:53

        from sentence_transformers import SentenceTransformer
    except ModuleNotFoundError:
        logger.error("The sentence_transformers module has not been found. Please install it manually with pip install -U sentence-transformers.")
        raise ModuleNotFoundError

    initialize_embedding_params()
    global embeddings_device, embeddings_model
    try:
        logger.info(f"Try embedding model: {model} on {embeddings_device}")
        if 'jina-embeddings' in model:
            embeddings_model = AutoModel.from_pretrained(model, trust_remote_code=shared.args.trust_remote_code)
            embeddings_model = embeddings_model.to(embeddings_device)
        else:
            embeddings_model = SentenceTransformer(model, device=embeddings_device)

        logger.info(f"Loaded embedding model: {model}")
    except Exception as e:
        embeddings_model = None
        raise ServiceUnavailableError(f"Error: Failed to load embedding model: {model}", internal_message=repr(e))


def get_embeddings_model():
    initialize_embedding_params()
    global embeddings_model, st_model
    if st_model and not embeddings_model:
        load_embedding_model(st_model)  # lazy load the model

    return embeddings_model


def get_embeddings_model_name() -> str:
    initialize_embedding_params()
    global st_model
    return st_model


def get_embeddings(input: list) -> np.ndarray:

View on GitHub (pinned to ed888c71f2)

Solutions

1. Check server logs for internal_message=repr(e) to see the underlying cause (HTTP 401/403, 404, CUDA error, ImportError).
2. Set a valid, public embedding model id (e.g. sentence-transformers/all-MiniLM-L6-v2) in the API embedding model setting.
3. For jina models, launch the server with --trust-remote-code.
4. Ensure the device exists: set embeddings device to cpu if no GPU, or fix CUDA install.
5. Pre-download the model (huggingface-cli download <model>) or point to a local snapshot path when offline.

Example fix

# before: server started with missing/gated model, embeddings calls 503
python server.py --api --embeddings-model somemodel/all-MiniLM

# after: valid public model on cpu
python server.py --api --embeddings-model sentence-transformers/all-MiniLM-L6-v2 --embeddings-device cpu

def embeddings_ready(base_url: str, model: str) -> bool:
    import requests
    r = requests.get(f'{base_url}/v1/internal/model/info', timeout=10)
    return r.ok and any(model in m.get('id', '') for m in r.json().get('data', []))

try:
    emb = client.embeddings.create(model=model, input=text)
except openai.APIStatusError as e:
    if e.status_code == 503 and 'Failed to load embedding model' in str(e):
        # check server logs for internal_message; fix config, then retry once after fixing
        raise RuntimeError(f'Embedding model {model} failed to load; see server log')
    raise

When it happens

Trigger: POST /v1/embeddings when the model configured (env HF_TOKEN gated model, non-existent repo id, or a local path that is absent) fails to download/load; CUDA device mismatch (embeddings_device=cuda with no GPU); a jina model loaded without --trust-remote-code; offline machine with no cached snapshot.

Common situations: Wrong or misspelled model name in the --api embedding model setting; network/proxy blocked HF downloads; no HF token for gated models (e.g. some sentence-transformers variants); sentence_transformers/transformers version conflict; disk-full during model download.

Related errors

• No image model loaded. Load a model via the UI first.
• Invalid branch name. Only alphanumeric characters, period, u
• Image generation failed or produced no images.
• No model specified.

AI-assisted analysis of oobabooga/textgen@ed888c71f2 (2026-08-15). Data as JSON: /api/errors/53ea92e2c9deb057. Report an issue: GitHub.