{"record":{"id":"cd54a47a5e1bc0aa","repo":"unslothai/unsloth","slug":"unsloth-distributed-mlx-inference-requires-parall","errorCode":null,"errorMessage":"Unsloth: distributed MLX inference requires parallel_mode='pipeline' or parallel_mode='tensor'.","messagePattern":"Unsloth: distributed MLX inference requires parallel_mode='pipeline' or parallel_mode='tensor'\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/inference/mlx_inference.py","lineNumber":1172,"sourceCode":"        if hf_token:\n            import os\n            os.environ[\"HF_TOKEN\"] = hf_token\n        self._configure_memory_limits()\n\n        is_lora = getattr(config, \"is_lora\", False)\n\n        logger.info(\n            \"Loading %s via %s (is_lora=%s, distributed=%s, rank=%s/%s, mode=%s)\",\n            model_name,\n            \"mlx-vlm\" if is_vision else \"mlx-lm\",\n            is_lora,\n            is_distributed,\n            distributed_rank,\n            distributed_size,\n            parallel_mode,\n        )\n        if is_distributed and parallel_mode not in (\"pipeline\", \"tensor\"):\n            raise ValueError(\n                \"Unsloth: distributed MLX inference requires parallel_mode='pipeline' \"\n                \"or parallel_mode='tensor'.\"\n            )\n        if is_distributed and is_lora:\n            raise ValueError(\n                \"Unsloth: distributed MLX inference for LoRA adapter repos \"\n                \"is not supported yet. Merge/export the adapter into an MLX model \"\n                \"before distributed inference.\"\n            )\n\n        try:\n            from unsloth_zoo.mlx.loader import FastMLXModel\n        except ImportError as e:\n            raise ImportError(\n                \"Unsloth: MLX inference requires unsloth-zoo with the MLX modules \"\n                \"(unsloth_zoo.mlx.loader). Reinstall via install.sh on Apple Silicon.\"\n            ) from e\n","sourceCodeStart":1154,"sourceCodeEnd":1190,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/inference/mlx_inference.py#L1154-L1190","documentation":"Validation in the MLX backend's distributed loading path: when a distributed group with world_size > 1 is active, parallel_mode must be either 'pipeline' or 'tensor'. These are the only two parallelism strategies the MLX orchestration implements, so any other value (or None) with multiple ranks is rejected up front with a ValueError before FastMLXModel is imported and ranks try to coordinate.","triggerScenarios":"Calling load with distributed_group set (world_size > 1) while parallel_mode is None, '', 'data', 'hybrid', or any unsupported string. is_distributed is computed as distributed_group is not None and distributed_size > 1, so single-rank setups never hit this.","commonSituations":"Defaulting parallel_mode to None in orchestrator config and forgetting to set it when launching multi-worker MLX; copy-pasting a single-process launch script to a multi-rank context; typos like 'Tensor' (case-sensitive).","solutions":["Set parallel_mode='pipeline' or parallel_mode='tensor' explicitly when launching with more than one rank.","If you did not intend distributed inference, ensure distributed_group is None or world_size == 1 in the loader call.","Check orchestrator/env config that computes distributed rank/size for a stale leftover (e.g. leftover RANK/WORLD_SIZE env vars from a previous distributed run)."],"exampleFix":"# before\nbackend.load('meta-llama/Llama-3.1-8B-Instruct-4bit',\n    distributed_group=group)  # parallel_mode defaults to None -> ValueError\n\n# after\nbackend.load('meta-llama/Llama-3.1-8B-Instruct-4bit',\n    distributed_group=group, parallel_mode='pipeline')","handlingStrategy":"validation","validationCode":"if distributed_size > 1:\n    assert parallel_mode in ('pipeline', 'tensor'), (\n        f\"parallel_mode must be 'pipeline' or 'tensor', got {parallel_mode!r}\")","typeGuard":"def is_valid_parallel_mode(mode: object) -> TypeGuard[str]:\n    return isinstance(mode, str) and mode in ('pipeline', 'tensor')","tryCatchPattern":"try:\n    backend.load(name, distributed_group=group, parallel_mode=mode)\nexcept ValueError as e:\n    if 'parallel_mode' in str(e):\n        mode = 'pipeline'\n        backend.load(name, distributed_group=group, parallel_mode=mode)\n    else:\n        raise","preventionTips":["Centralize launch config: derive parallel_mode from world_size in one place so multi-rank launches always set it.","Validate config at startup, not per-request.","Use exact lowercase strings; the check is case-sensitive."],"tags":["mlx","distributed","parallelism","config"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}