{"record":{"id":"502ac2c65a63c0f8","repo":"unslothai/unsloth","slug":"mlxinferencebackend-cannot-load-gguf-model-model","errorCode":null,"errorMessage":"MLXInferenceBackend cannot load GGUF model '{model_name}': GGUF models must be served by llama-server in the parent process. The /api/inference/load route should have detected this repo as GGUF before dispatching to the MLX orchestrator -- this fallback indicates a transient HF Hub failure during initial detection. Retry the request.","messagePattern":"MLXInferenceBackend cannot load GGUF model '(.+?)': GGUF models must be served by llama-server in the parent process\\. The /api/inference/load route should have detected this repo as GGUF before dispatching to the MLX orchestrator -- this fallback indicates a transient HF Hub failure during initial detection\\. Retry the request\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/inference/mlx_inference.py","lineNumber":1145,"sourceCode":"        import mlx.core as mx\n\n        # Keep the token so the native-template fallback can fetch a gated\n        # model's repo template during generation.\n        self._hf_token = hf_token\n        model_name = config.identifier if hasattr(config, \"identifier\") else str(config)\n        is_vision = getattr(config, \"is_vision\", False)\n        distributed_rank, distributed_size = _mlx_distributed_rank_size(distributed_group)\n        is_distributed = distributed_group is not None and distributed_size > 1\n        self._distributed_group = distributed_group\n        self._distributed_rank = distributed_rank\n        self._distributed_world_size = distributed_size\n\n        # GGUF guard: GGUF is served by llama-server in the parent process,\n        # not mlx-lm. Reaching here with is_gguf=True means the route's\n        # detection flaked but the subprocess re-detected GGUF; raise loudly\n        # instead of a cryptic mlx_lm error.\n        if getattr(config, \"is_gguf\", False):\n            raise RuntimeError(\n                f\"MLXInferenceBackend cannot load GGUF model '{model_name}': \"\n                f\"GGUF models must be served by llama-server in the parent \"\n                f\"process. The /api/inference/load route should have \"\n                f\"detected this repo as GGUF before dispatching to the MLX \"\n                f\"orchestrator -- this fallback indicates a transient HF \"\n                f\"Hub failure during initial detection. Retry the request.\"\n            )\n\n        if hf_token:\n            import os\n            os.environ[\"HF_TOKEN\"] = hf_token\n        self._configure_memory_limits()\n\n        is_lora = getattr(config, \"is_lora\", False)\n\n        logger.info(\n            \"Loading %s via %s (is_lora=%s, distributed=%s, rank=%s/%s, mode=%s)\",\n            model_name,","sourceCodeStart":1127,"sourceCodeEnd":1163,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/inference/mlx_inference.py#L1127-L1163","documentation":"Guard inside MLXInferenceBackend.load: GGUF-quantized models cannot be loaded by mlx-lm — they are served by llama-server in the parent process. The /api/inference/load route is supposed to detect GGUF repos and dispatch to llama-server; reaching the MLX loader with config.is_gguf=True means the route's Hub detection flaked (e.g. a transient HF Hub failure) and the subprocess re-detected GGUF. The loud RuntimeError replaces a cryptic mlx_lm parse error, and the message explicitly says retrying is the remedy.","triggerScenarios":"Loading a GGUF repo (e.g. *-Q4_K_M.gguf files) through /api/inference/load while HF Hub briefly fails so the route's GGUF detection misses it, then the MLX orchestrator subprocess re-checks config and finds is_gguf=True.","commonSituations":"HF Hub timeouts/503s or flaky network on first model resolution; racing Hub metadata during load; retrying immediately after the transient failure typically routes correctly to llama-server.","solutions":["Retry the load request — the transient HF Hub failure usually clears and the route then dispatches to llama-server.","If it persists, check HF Hub reachability (hub outage, token, proxy) and pre-warm detection with a direct huggingface_hub call for the repo.","Confirm the repo really is GGUF and not mis-detected; if the intended backend is MLX, use an MLX-quantized (mlx-format) repo instead."],"exampleFix":"# before\nresp = client.post('/api/inference/load', json={'model': 'user/model-GGUF'})  # transient hub flake\n\n# after\nfor attempt in range(3):\n    resp = client.post('/api/inference/load', json={'model': 'user/model-GGUF'})\n    if resp.status_code != 502:\n        break\n    time.sleep(2 ** attempt)  # retry transient HF Hub failure","handlingStrategy":"retry","validationCode":"info = huggingface_hub.model_info(model_name)  # or cached GGUF probe\ngguf = any(f.rfilename.endswith('.gguf') for f in info.siblings)\nif gguf:\n    route_to_llama_server(model_name)  # never dispatch GGUF to MLX","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    try:\n        return await load_model(model_name)\n    except RuntimeError as e:\n        if 'GGUF' in str(e) and 'Retry' in str(e) and attempt < 2:\n            await asyncio.sleep(2 ** attempt)\n            continue\n        raise","preventionTips":["Make GGUF detection deterministic (cache the repo file list) so transient Hub failures cannot mis-route.","Treat this error as a signal to retry, not to reconfigure — the message explicitly says retry.","Add monitoring on how often the fallback fires; frequent hits mean Hub connectivity problems worth fixing."],"tags":["gguf","llama-server","huggingface-hub","transient","model-loading"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}