unslothai/unsloth · error · ValueError

Unsloth: distributed MLX inference requires parallel_mode='p

Error message

Unsloth: distributed MLX inference requires parallel_mode='pipeline' or parallel_mode='tensor'.

What it means

Validation in the MLX backend's distributed loading path: when a distributed group with world_size > 1 is active, parallel_mode must be either 'pipeline' or 'tensor'. These are the only two parallelism strategies the MLX orchestration implements, so any other value (or None) with multiple ranks is rejected up front with a ValueError before FastMLXModel is imported and ranks try to coordinate.

Source

Thrown at studio/backend/core/inference/mlx_inference.py:1172

        if hf_token:
            import os
            os.environ["HF_TOKEN"] = hf_token
        self._configure_memory_limits()

        is_lora = getattr(config, "is_lora", False)

        logger.info(
            "Loading %s via %s (is_lora=%s, distributed=%s, rank=%s/%s, mode=%s)",
            model_name,
            "mlx-vlm" if is_vision else "mlx-lm",
            is_lora,
            is_distributed,
            distributed_rank,
            distributed_size,
            parallel_mode,
        )
        if is_distributed and parallel_mode not in ("pipeline", "tensor"):
            raise ValueError(
                "Unsloth: distributed MLX inference requires parallel_mode='pipeline' "
                "or parallel_mode='tensor'."
            )
        if is_distributed and is_lora:
            raise ValueError(
                "Unsloth: distributed MLX inference for LoRA adapter repos "
                "is not supported yet. Merge/export the adapter into an MLX model "
                "before distributed inference."
            )

        try:
            from unsloth_zoo.mlx.loader import FastMLXModel
        except ImportError as e:
            raise ImportError(
                "Unsloth: MLX inference requires unsloth-zoo with the MLX modules "
                "(unsloth_zoo.mlx.loader). Reinstall via install.sh on Apple Silicon."
            ) from e

View on GitHub (pinned to 203007d190)

Solutions

Set parallel_mode='pipeline' or parallel_mode='tensor' explicitly when launching with more than one rank.
If you did not intend distributed inference, ensure distributed_group is None or world_size == 1 in the loader call.
Check orchestrator/env config that computes distributed rank/size for a stale leftover (e.g. leftover RANK/WORLD_SIZE env vars from a previous distributed run).

Example fix

# before
backend.load('meta-llama/Llama-3.1-8B-Instruct-4bit',
    distributed_group=group)  # parallel_mode defaults to None -> ValueError

# after
backend.load('meta-llama/Llama-3.1-8B-Instruct-4bit',
    distributed_group=group, parallel_mode='pipeline')

Defensive patterns

Strategy: validation

Validate before calling

if distributed_size > 1:
    assert parallel_mode in ('pipeline', 'tensor'), (
        f"parallel_mode must be 'pipeline' or 'tensor', got {parallel_mode!r}")

Type guard

def is_valid_parallel_mode(mode: object) -> TypeGuard[str]:
    return isinstance(mode, str) and mode in ('pipeline', 'tensor')

Try / catch

try:
    backend.load(name, distributed_group=group, parallel_mode=mode)
except ValueError as e:
    if 'parallel_mode' in str(e):
        mode = 'pipeline'
        backend.load(name, distributed_group=group, parallel_mode=mode)
    else:
        raise

Prevention

Centralize launch config: derive parallel_mode from world_size in one place so multi-rank launches always set it.
Validate config at startup, not per-request.
Use exact lowercase strings; the check is case-sensitive.

When it happens

Trigger: Calling load with distributed_group set (world_size > 1) while parallel_mode is None, '', 'data', 'hybrid', or any unsupported string. is_distributed is computed as distributed_group is not None and distributed_size > 1, so single-rank setups never hit this.

Common situations: Defaulting parallel_mode to None in orchestrator config and forgetting to set it when launching multi-worker MLX; copy-pasting a single-process launch script to a multi-rank context; typos like 'Tensor' (case-sensitive).

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/cd54a47a5e1bc0aa. Report an issue: GitHub.