{"record":{"id":"2e65444b59d3efbb","repo":"unslothai/unsloth","slug":"unsloth-distributed-mlx-inference-for-lora-adapte","errorCode":null,"errorMessage":"Unsloth: distributed MLX inference for LoRA adapter repos is not supported yet. Merge/export the adapter into an MLX model before distributed inference.","messagePattern":"Unsloth: distributed MLX inference for LoRA adapter repos is not supported yet\\. Merge/export the adapter into an MLX model before distributed inference\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/inference/mlx_inference.py","lineNumber":1177,"sourceCode":"        is_lora = getattr(config, \"is_lora\", False)\n\n        logger.info(\n            \"Loading %s via %s (is_lora=%s, distributed=%s, rank=%s/%s, mode=%s)\",\n            model_name,\n            \"mlx-vlm\" if is_vision else \"mlx-lm\",\n            is_lora,\n            is_distributed,\n            distributed_rank,\n            distributed_size,\n            parallel_mode,\n        )\n        if is_distributed and parallel_mode not in (\"pipeline\", \"tensor\"):\n            raise ValueError(\n                \"Unsloth: distributed MLX inference requires parallel_mode='pipeline' \"\n                \"or parallel_mode='tensor'.\"\n            )\n        if is_distributed and is_lora:\n            raise ValueError(\n                \"Unsloth: distributed MLX inference for LoRA adapter repos \"\n                \"is not supported yet. Merge/export the adapter into an MLX model \"\n                \"before distributed inference.\"\n            )\n\n        try:\n            from unsloth_zoo.mlx.loader import FastMLXModel\n        except ImportError as e:\n            raise ImportError(\n                \"Unsloth: MLX inference requires unsloth-zoo with the MLX modules \"\n                \"(unsloth_zoo.mlx.loader). Reinstall via install.sh on Apple Silicon.\"\n            ) from e\n\n        load_kwargs = {\n            \"max_seq_length\": max_seq_length,\n            \"dtype\": dtype,\n            \"load_in_4bit\": load_in_4bit,\n            \"token\": hf_token,","sourceCodeStart":1159,"sourceCodeEnd":1195,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/inference/mlx_inference.py#L1159-L1195","documentation":"The distributed MLX path explicitly refuses LoRA adapter repos: with world_size > 1 and is_lora=True, loading is rejected because MLX pipeline/tensor parallel inference over an unmerged adapter repo is not implemented. The error tells you the supported route — merge or export the adapter into a full MLX model first, then run distributed inference on the merged checkpoint.","triggerScenarios":"Calling load with distributed_group world_size > 1 on a repo detected as a LoRA adapter (adapter_config.json + adapter weights, no full base weights).","commonSituations":"Trying to serve a fine-tuned LoRA (e.g. from an unsloth training run) across multiple Macs; assuming adapter hot-swap works in distributed mode the way it does single-node.","solutions":["Merge/export the adapter into a full MLX model (unsloth merge/export tooling), push or point to the merged repo, and load that with parallel_mode set.","Alternatively serve the adapter single-rank (no distributed_group) where adapter loading is supported.","Verify the repo you passed is actually the merged model, not the adapter-only repo — detection keys off adapter markers in the repo."],"exampleFix":"# before\nbackend.load('me/my-lora-adapter', distributed_group=group, parallel_mode='pipeline')\n\n# after\n# merge first: unsloth merge -> 'me/my-lora-merged-mlx'\nbackend.load('me/my-lora-merged-mlx', distributed_group=group, parallel_mode='pipeline')","handlingStrategy":"validation","validationCode":"if distributed_size > 1 and repo_is_lora(model_name):\n    raise ValueError('merge the adapter before distributed inference')\n# route to merge/export pipeline instead of the MLX loader","typeGuard":null,"tryCatchPattern":"try:\n    backend.load(repo, distributed_group=group, parallel_mode='pipeline')\nexcept ValueError as e:\n    if 'LoRA adapter repos' in str(e):\n        merged = merge_adapter(repo)  # export pipeline\n        backend.load(merged, distributed_group=group, parallel_mode='pipeline')\n    else:\n        raise","preventionTips":["Detect adapter repos (adapter_config.json) before distributed launch and auto-route to the merge/export pipeline.","Document that distributed mode requires full (merged) MLX checkpoints.","In CI, run a distributed smoke-load on every supported model type to catch routing gaps."],"tags":["mlx","lora","distributed","model-merging"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}