unslothai/unsloth · error · ImportError

Unsloth: MLX inference requires unsloth-zoo with the MLX mod

Error message

Unsloth: MLX inference requires unsloth-zoo with the MLX modules (unsloth_zoo.mlx.loader). Reinstall via install.sh on Apple Silicon.

What it means

ImportError raised when `from unsloth_zoo.mlx.loader import FastMLXModel` fails inside the MLX backend's load path. The MLX inference stack lives in unsloth-zoo's MLX modules, which are only installed by install.sh on Apple Silicon; a Linux install, a pip install of unsloth-zoo without the MLX extras, or a broken environment makes the import fail. The original ImportError is chained as the cause.

Source

Thrown at studio/backend/core/inference/mlx_inference.py:1186

            distributed_size,
            parallel_mode,
        )
        if is_distributed and parallel_mode not in ("pipeline", "tensor"):
            raise ValueError(
                "Unsloth: distributed MLX inference requires parallel_mode='pipeline' "
                "or parallel_mode='tensor'."
            )
        if is_distributed and is_lora:
            raise ValueError(
                "Unsloth: distributed MLX inference for LoRA adapter repos "
                "is not supported yet. Merge/export the adapter into an MLX model "
                "before distributed inference."
            )

        try:
            from unsloth_zoo.mlx.loader import FastMLXModel
        except ImportError as e:
            raise ImportError(
                "Unsloth: MLX inference requires unsloth-zoo with the MLX modules "
                "(unsloth_zoo.mlx.loader). Reinstall via install.sh on Apple Silicon."
            ) from e

        load_kwargs = {
            "max_seq_length": max_seq_length,
            "dtype": dtype,
            "load_in_4bit": load_in_4bit,
            "token": hf_token,
            "trust_remote_code": trust_remote_code,
            "text_only": False if is_vision else True,
        }
        if is_distributed:
            if parallel_mode == "pipeline":
                load_kwargs["pipeline_group"] = distributed_group
            else:
                load_kwargs["tensor_group"] = distributed_group

View on GitHub (pinned to 203007d190)

Solutions

Run the project's install.sh on an Apple Silicon macOS machine — it installs unsloth-zoo with the MLX modules.
Verify the module resolves: python -c "from unsloth_zoo.mlx.loader import FastMLXModel" and fix the environment it fails in.
On non-Apple-Silicon hosts, use a supported backend (llama-server/GGUF or transformers) instead of the MLX orchestrator.

Example fix

# before (linux / incomplete env)
backend.load('mlx-community/Llama-3.1-8B-Instruct-4bit')  # ImportError

# after
# on Apple Silicon macOS:
# ./install.sh  (installs unsloth-zoo with MLX modules)
backend.load('mlx-community/Llama-3.1-8B-Instruct-4bit')

Defensive patterns

Strategy: validation

Validate before calling

def mlx_backend_available() -> bool:
    try:
        import unsloth_zoo.mlx.loader  # noqa: F401
        return True
    except ImportError:
        return False

if not mlx_backend_available():
    disable_route('mlx')  # or fall back to transformers/llama-server

Try / catch

try:
    backend.load(model_name)
except ImportError as e:
    if 'unsloth_zoo.mlx.loader' in str(e):
        raise RuntimeError('MLX backend unavailable on this host; use llama-server backend') from e
    raise

Prevention

Feature-probe for unsloth_zoo.mlx.loader at service startup and hide/disable the MLX route on hosts without it.
Provision Apple Silicon environments exclusively through install.sh so MLX modules are always present.
Fail fast in deployment checks rather than at first model load.

When it happens

Trigger: Loading any model through the MLX backend on a machine where unsloth_zoo.mlx.loader is absent: plain pip install unsloth-zoo (no MLX modules), Linux/x86 hosts, or a partially failed install.sh run.

Common situations: Running the studio backend on Linux where it fell back to the MLX orchestrator; mixed conda/venv environments where unsloth-zoo was installed without MLX deps; Apple Silicon box where install.sh was interrupted or the venv was recreated by hand.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/f28028d44bfae808. Report an issue: GitHub.