mudler/LocalAI · error · FileNotFoundError

Expected HF model directory, got file

Error message

Expected HF model directory, got file: {model_path}

What it means

In the LLM constructor's HF-safetensors branch (taken when the model ref is a directory without a GGUF file), model_path must be a directory containing an HF layout. If the path exists but is a regular file that is not .gguf (e.g. a single .safetensors or .bin file), the branch refuses it with FileNotFoundError.

Solutions

1. Pass the directory that contains config.json and the safetensors files, not an individual weight file.
2. If you only have a single GGUF file, keep using it — that path is supported; for HF safetensors the directory layout is required.
3. Re-download/restore the full HF repo layout (config.json + tokenizer + weights) into a directory and point the model at it.

Example fix

# before
model = TinyGradLLM(model_path=Path("/models/qwen/model.safetensors"))
# after
model = TinyGradLLM(model_path=Path("/models/qwen"))  # dir with config.json

from pathlib import Path

def is_hf_model_dir(p) -> bool:
    p = Path(p)
    return p.is_dir() and (p / "config.json").exists()

def is_hf_model_dir(p) -> bool:
    p = Path(p)
    return p.is_dir() and (p / "config.json").exists()

try:
    llm = LLM(model_path)
except FileNotFoundError as e:
    if "Expected HF model directory" in str(e):
        model_path = Path(model_path).parent  # only if parent is the real repo root
    raise

When it happens

Trigger: Passing a single weight file such as /models/qwen/model.safetensors as the model path instead of its parent directory; pointing at a symlinked file; passing a .pt or .bin checkpoint where the GGUF branch is not taken either.

Common situations: User extracts one downloaded file and passes it directly; misconfigured gallery entry with a filename instead of directory; conflating GGUF single-file usage (supported) with safetensors single-file usage (not supported).

Related errors

• config.json not found under
• Model not found
• tokenizer.json not found under
• No safetensors weights found under
• ONNX model not found

AI-assisted analysis of mudler/LocalAI@44413a9d06 (2026-08-15). Data as JSON: /api/errors/1b82cbf89af9091d. Report an issue: GitHub.

Appendix: source

Thrown at backend/python/tinygrad/backend.py:315

            model, kv = Transformer.from_gguf(gguf_tensor, max_context=max_context_cap)
            self.llm_model = model
            self.max_context = model.max_context
            # Preserve a config-shaped dict for tool-parser heuristics and
            # the "loaded" message.
            arch = kv.get("general.architecture", "")
            self.llm_config = {
                "architectures": [kv.get("general.name", arch) or arch],
                "gguf_kv": kv,
            }

            # Tokenizer: prefer sidecar tokenizer.json (richer HF Jinja2
            # templates), fall back to apps.llm's SimpleTokenizer built
            # from GGUF metadata.
            self._load_tokenizer_for_dir(model_path if model_path.is_dir() else gguf_file.parent, gguf_kv=kv)
        else:
            # HF safetensors path.
            if not model_path.is_dir():
                raise FileNotFoundError(f"Expected HF model directory, got file: {model_path}")
            config_path = model_path / "config.json"
            if not config_path.exists():
                raise FileNotFoundError(f"config.json not found under {model_path}")
            with open(config_path) as fp:
                hf_config = json.load(fp)
            self.llm_config = hf_config

            raw_weights = _load_hf_safetensors(model_path)
            n_layers = hf_config["num_hidden_layers"]
            state_dict = _hf_to_appsllm_state_dict(raw_weights, n_layers)

            kwargs = _hf_to_transformer_kwargs(hf_config, state_dict, max_context_cap)
            self.max_context = kwargs["max_context"]

            model = Transformer(**kwargs)
            load_state_dict(model, state_dict, strict=False, consume=True)
            self.llm_model = model

View on GitHub (pinned to 44413a9d06)