mudler/LocalAI · error · ValueError

Model must have a 'model' or 'transformer' attribute

Error message

Model must have a 'model' or 'transformer' attribute

What it means

Raised by get_inner_model() in mlx-distributed/sharding.py when the wrapped model exposes neither a 'model' nor a 'transformer' attribute that is an nn.Module. The sharding code needs the inner transformer stack to slice layers across ranks, so it walks two common attribute names and fails if neither matches the model's structure.

Source

Thrown at backend/python/mlx-distributed/sharding.py:85

        # Gather output from all ranks so every rank has the final result
        output = mx.distributed.all_gather(output, group=self.group)[
            -output.shape[0] :
        ]
        mx.eval(output)
        return output


def get_inner_model(model):
    """Get the inner model (model.model or model.transformer)."""
    for attr in ("model", "transformer"):
        inner = getattr(model, attr, None)
        if isinstance(inner, nn.Module):
            # Some models have model.model (e.g. language_model.model)
            inner_inner = getattr(inner, "model", None)
            if isinstance(inner_inner, nn.Module):
                return inner_inner
            return inner
    raise ValueError("Model must have a 'model' or 'transformer' attribute")


def get_layers(inner_model):
    """Get the list of transformer layers."""
    for attr in ("layers", "h"):
        layers = getattr(inner_model, attr, None)
        if layers is not None:
            return layers
    raise ValueError("Model must have a 'layers' or 'h' attribute")


def pipeline_auto_parallel(model, group, start_layer=None, end_layer=None):
    """Apply pipeline parallelism to a model.

    Each rank only keeps its slice of layers.  The first layer receives from
    the previous rank, and the last layer sends to the next rank.

    Args:

View on GitHub (pinned to 44413a9d06)

Solutions

1. Inspect the model with vars(model)/dir(model) to find the real attribute holding the transformer stack
2. Wrap or alias: model.model = <inner module> before sharding
3. Extend the ('model', 'transformer') tuple in get_inner_model with the attribute your architecture uses (and upstream it)

Example fix

# before
inner = get_inner_model(model)  # ValueError: must have 'model' or 'transformer'

# after
# for models exposing 'language_model'
model.model = model.language_model
inner = get_inner_model(model)

import mlx.nn as nn
candidates = [getattr(model, a, None) for a in ('model', 'transformer')]
if not any(isinstance(c, nn.Module) for c in candidates):
    raise ValueError(f'unsupported architecture {type(model).__name__}: cannot locate inner transformer; attrs={ [k for k,v in vars(model).items() if isinstance(v, nn.Module)] }')

def has_inner_module(model) -> bool:
    return any(isinstance(getattr(model, a, None), nn.Module) for a in ('model', 'transformer'))

try:
    inner = get_inner_model(model)
except ValueError as err:
    raise RuntimeError(f'cannot shard {type(model).__name__}: {err}') from err

When it happens

Trigger: Loading a model class that nests its layers differently (e.g. attribute named 'language_model', 'backbone', or layers directly on the top-level object) and then calling pipeline_auto_parallel/get_inner_model on it.

Common situations: New or unusual HF-compatible MLX model architectures whose internals don't follow the model.model/model.transformer convention; mlx-lm version changes renaming attributes; custom model wrappers.

Related errors

• Model must have a 'layers' or 'h' attribute
• Unknown backend: {backend}
• onnx_direct engine requires both detector_onnx and recognize
• unknown engine: {name!r}
• Model not loaded

AI-assisted analysis of mudler/LocalAI@44413a9d06 (2026-08-15). Data as JSON: /api/errors/2478842a15eee665. Report an issue: GitHub.