unslothai/unsloth · error · RuntimeError

MLXInferenceBackend cannot load GGUF model '{model_name}': G

Error message

MLXInferenceBackend cannot load GGUF model '{model_name}': GGUF models must be served by llama-server in the parent process. The /api/inference/load route should have detected this repo as GGUF before dispatching to the MLX orchestrator -- this fallback indicates a transient HF Hub failure during initial detection. Retry the request.

What it means

Guard inside MLXInferenceBackend.load: GGUF-quantized models cannot be loaded by mlx-lm — they are served by llama-server in the parent process. The /api/inference/load route is supposed to detect GGUF repos and dispatch to llama-server; reaching the MLX loader with config.is_gguf=True means the route's Hub detection flaked (e.g. a transient HF Hub failure) and the subprocess re-detected GGUF. The loud RuntimeError replaces a cryptic mlx_lm parse error, and the message explicitly says retrying is the remedy.

Source

Thrown at studio/backend/core/inference/mlx_inference.py:1145

        import mlx.core as mx

        # Keep the token so the native-template fallback can fetch a gated
        # model's repo template during generation.
        self._hf_token = hf_token
        model_name = config.identifier if hasattr(config, "identifier") else str(config)
        is_vision = getattr(config, "is_vision", False)
        distributed_rank, distributed_size = _mlx_distributed_rank_size(distributed_group)
        is_distributed = distributed_group is not None and distributed_size > 1
        self._distributed_group = distributed_group
        self._distributed_rank = distributed_rank
        self._distributed_world_size = distributed_size

        # GGUF guard: GGUF is served by llama-server in the parent process,
        # not mlx-lm. Reaching here with is_gguf=True means the route's
        # detection flaked but the subprocess re-detected GGUF; raise loudly
        # instead of a cryptic mlx_lm error.
        if getattr(config, "is_gguf", False):
            raise RuntimeError(
                f"MLXInferenceBackend cannot load GGUF model '{model_name}': "
                f"GGUF models must be served by llama-server in the parent "
                f"process. The /api/inference/load route should have "
                f"detected this repo as GGUF before dispatching to the MLX "
                f"orchestrator -- this fallback indicates a transient HF "
                f"Hub failure during initial detection. Retry the request."
            )

        if hf_token:
            import os
            os.environ["HF_TOKEN"] = hf_token
        self._configure_memory_limits()

        is_lora = getattr(config, "is_lora", False)

        logger.info(
            "Loading %s via %s (is_lora=%s, distributed=%s, rank=%s/%s, mode=%s)",
            model_name,

View on GitHub (pinned to 203007d190)

Solutions

Retry the load request — the transient HF Hub failure usually clears and the route then dispatches to llama-server.
If it persists, check HF Hub reachability (hub outage, token, proxy) and pre-warm detection with a direct huggingface_hub call for the repo.
Confirm the repo really is GGUF and not mis-detected; if the intended backend is MLX, use an MLX-quantized (mlx-format) repo instead.

Example fix

# before
resp = client.post('/api/inference/load', json={'model': 'user/model-GGUF'})  # transient hub flake

# after
for attempt in range(3):
    resp = client.post('/api/inference/load', json={'model': 'user/model-GGUF'})
    if resp.status_code != 502:
        break
    time.sleep(2 ** attempt)  # retry transient HF Hub failure

Defensive patterns

Strategy: retry

Validate before calling

info = huggingface_hub.model_info(model_name)  # or cached GGUF probe
gguf = any(f.rfilename.endswith('.gguf') for f in info.siblings)
if gguf:
    route_to_llama_server(model_name)  # never dispatch GGUF to MLX

Try / catch

for attempt in range(3):
    try:
        return await load_model(model_name)
    except RuntimeError as e:
        if 'GGUF' in str(e) and 'Retry' in str(e) and attempt < 2:
            await asyncio.sleep(2 ** attempt)
            continue
        raise

Prevention

Make GGUF detection deterministic (cache the repo file list) so transient Hub failures cannot mis-route.
Treat this error as a signal to retry, not to reconfigure — the message explicitly says retry.
Add monitoring on how often the fallback fires; frequent hits mean Hub connectivity problems worth fixing.

When it happens

Trigger: Loading a GGUF repo (e.g. *-Q4_K_M.gguf files) through /api/inference/load while HF Hub briefly fails so the route's GGUF detection misses it, then the MLX orchestrator subprocess re-checks config and finds is_gguf=True.

Common situations: HF Hub timeouts/503s or flaky network on first model resolution; racing Hub metadata during load; retrying immediately after the transient failure typically routes correctly to llama-server.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/502ac2c65a63c0f8. Report an issue: GitHub.