unslothai/unsloth · error · ValueError

Unsloth: distributed MLX inference for LoRA adapter repos is

Error message

Unsloth: distributed MLX inference for LoRA adapter repos is not supported yet. Merge/export the adapter into an MLX model before distributed inference.

What it means

The distributed MLX path explicitly refuses LoRA adapter repos: with world_size > 1 and is_lora=True, loading is rejected because MLX pipeline/tensor parallel inference over an unmerged adapter repo is not implemented. The error tells you the supported route — merge or export the adapter into a full MLX model first, then run distributed inference on the merged checkpoint.

Source

Thrown at studio/backend/core/inference/mlx_inference.py:1177

        is_lora = getattr(config, "is_lora", False)

        logger.info(
            "Loading %s via %s (is_lora=%s, distributed=%s, rank=%s/%s, mode=%s)",
            model_name,
            "mlx-vlm" if is_vision else "mlx-lm",
            is_lora,
            is_distributed,
            distributed_rank,
            distributed_size,
            parallel_mode,
        )
        if is_distributed and parallel_mode not in ("pipeline", "tensor"):
            raise ValueError(
                "Unsloth: distributed MLX inference requires parallel_mode='pipeline' "
                "or parallel_mode='tensor'."
            )
        if is_distributed and is_lora:
            raise ValueError(
                "Unsloth: distributed MLX inference for LoRA adapter repos "
                "is not supported yet. Merge/export the adapter into an MLX model "
                "before distributed inference."
            )

        try:
            from unsloth_zoo.mlx.loader import FastMLXModel
        except ImportError as e:
            raise ImportError(
                "Unsloth: MLX inference requires unsloth-zoo with the MLX modules "
                "(unsloth_zoo.mlx.loader). Reinstall via install.sh on Apple Silicon."
            ) from e

        load_kwargs = {
            "max_seq_length": max_seq_length,
            "dtype": dtype,
            "load_in_4bit": load_in_4bit,
            "token": hf_token,

View on GitHub (pinned to 203007d190)

Solutions

1. Merge/export the adapter into a full MLX model (unsloth merge/export tooling), push or point to the merged repo, and load that with parallel_mode set.
2. Alternatively serve the adapter single-rank (no distributed_group) where adapter loading is supported.
3. Verify the repo you passed is actually the merged model, not the adapter-only repo — detection keys off adapter markers in the repo.

Example fix

# before
backend.load('me/my-lora-adapter', distributed_group=group, parallel_mode='pipeline')

# after
# merge first: unsloth merge -> 'me/my-lora-merged-mlx'
backend.load('me/my-lora-merged-mlx', distributed_group=group, parallel_mode='pipeline')

if distributed_size > 1 and repo_is_lora(model_name):
    raise ValueError('merge the adapter before distributed inference')
# route to merge/export pipeline instead of the MLX loader

try:
    backend.load(repo, distributed_group=group, parallel_mode='pipeline')
except ValueError as e:
    if 'LoRA adapter repos' in str(e):
        merged = merge_adapter(repo)  # export pipeline
        backend.load(merged, distributed_group=group, parallel_mode='pipeline')
    else:
        raise

When it happens

Trigger: Calling load with distributed_group world_size > 1 on a repo detected as a LoRA adapter (adapter_config.json + adapter weights, no full base weights).

Common situations: Trying to serve a fine-tuned LoRA (e.g. from an unsloth training run) across multiple Macs; assuming adapter hot-swap works in distributed mode the way it does single-node.

Related errors

• Unsloth MLX: use_adapter must be None, True, False, or a str
• Unsloth MLX: cannot disable adapter layers without their bas
• Unsloth: distributed MLX inference requires parallel_mode='p
• The requested LoRA adapters could not be applied: baking ada
• GGUF LoRA adapters are not supported on the diffusers engine

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/2e65444b59d3efbb. Report an issue: GitHub.