mudler/LocalAI · error · ValueError

Model must have a 'layers' or 'h' attribute

Error message

Model must have a 'layers' or 'h' attribute

What it means

Raised by get_layers() in mlx-distributed/sharding.py when the inner model has neither a 'layers' nor an 'h' attribute. After locating the inner module, the pipeline-parallel code needs the layer list to compute each rank's slice, and these two names cover MLX/Llama-style stacks; anything else fails here.

Source

Thrown at backend/python/mlx-distributed/sharding.py:94

    """Get the inner model (model.model or model.transformer)."""
    for attr in ("model", "transformer"):
        inner = getattr(model, attr, None)
        if isinstance(inner, nn.Module):
            # Some models have model.model (e.g. language_model.model)
            inner_inner = getattr(inner, "model", None)
            if isinstance(inner_inner, nn.Module):
                return inner_inner
            return inner
    raise ValueError("Model must have a 'model' or 'transformer' attribute")


def get_layers(inner_model):
    """Get the list of transformer layers."""
    for attr in ("layers", "h"):
        layers = getattr(inner_model, attr, None)
        if layers is not None:
            return layers
    raise ValueError("Model must have a 'layers' or 'h' attribute")


def pipeline_auto_parallel(model, group, start_layer=None, end_layer=None):
    """Apply pipeline parallelism to a model.

    Each rank only keeps its slice of layers.  The first layer receives from
    the previous rank, and the last layer sends to the next rank.

    Args:
        model: The MLX model (must have model.layers or similar)
        group: The distributed group
        start_layer: First layer index for this rank (auto-computed if None)
        end_layer: Last layer index (exclusive) for this rank (auto-computed if None)
    """
    rank = group.rank()
    world_size = group.size()

    inner = get_inner_model(model)

View on GitHub (pinned to 44413a9d06)

Solutions

1. Inspect getattr(inner, attr) candidates and find the actual layer container
2. Add the attribute name to the ('layers', 'h') tuple in get_layers for your architecture
3. For non-layered architectures, pipeline parallelism does not apply — use another sharding strategy

Example fix

# before
layers = get_layers(inner)  # ValueError: must have 'layers' or 'h'

# after
# architecture stores blocks as 'blocks'
layers = inner.blocks
# or patch: for attr in ("layers", "h", "blocks")

inner = get_inner_model(model)
layer_attrs = [a for a in ('layers', 'h') if getattr(inner, a, None) is not None]
if not layer_attrs:
    raise ValueError(f'{type(inner).__name__} exposes no layer list; pipeline sharding unsupported (attrs={dir(inner)})')

def has_layer_list(inner) -> bool:
    return any(getattr(inner, a, None) is not None for a in ('layers', 'h'))

try:
    layers = get_layers(inner)
except ValueError as err:
    raise RuntimeError(f'pipeline parallelism unavailable for this model: {err}') from err

When it happens

Trigger: The inner model stores its transformer blocks under a different name (e.g. 'decoder.layers', 'blocks', 'layers_list') or is a Mamba/SSM model with no layered decoder at all, and pipeline_auto_parallel -> get_layers is called.

Common situations: Non-transformer or unusually structured architectures (Mamba, hybrid models), custom nn.Module containers, or mlx version drift renaming block collections.

Related errors

• Model must have a 'model' or 'transformer' attribute
• Unknown backend: {backend}
• onnx_direct engine requires both detector_onnx and recognize
• unknown engine: {name!r}
• Model not loaded

AI-assisted analysis of mudler/LocalAI@44413a9d06 (2026-08-15). Data as JSON: /api/errors/c72b5edf364ce5fe. Report an issue: GitHub.