{"record":{"id":"c72b5edf364ce5fe","repo":"mudler/LocalAI","slug":"model-must-have-a-layers-or-h-attribute","errorCode":null,"errorMessage":"Model must have a 'layers' or 'h' attribute","messagePattern":"Model must have a 'layers' or 'h' attribute","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"backend/python/mlx-distributed/sharding.py","lineNumber":94,"sourceCode":"    \"\"\"Get the inner model (model.model or model.transformer).\"\"\"\n    for attr in (\"model\", \"transformer\"):\n        inner = getattr(model, attr, None)\n        if isinstance(inner, nn.Module):\n            # Some models have model.model (e.g. language_model.model)\n            inner_inner = getattr(inner, \"model\", None)\n            if isinstance(inner_inner, nn.Module):\n                return inner_inner\n            return inner\n    raise ValueError(\"Model must have a 'model' or 'transformer' attribute\")\n\n\ndef get_layers(inner_model):\n    \"\"\"Get the list of transformer layers.\"\"\"\n    for attr in (\"layers\", \"h\"):\n        layers = getattr(inner_model, attr, None)\n        if layers is not None:\n            return layers\n    raise ValueError(\"Model must have a 'layers' or 'h' attribute\")\n\n\ndef pipeline_auto_parallel(model, group, start_layer=None, end_layer=None):\n    \"\"\"Apply pipeline parallelism to a model.\n\n    Each rank only keeps its slice of layers.  The first layer receives from\n    the previous rank, and the last layer sends to the next rank.\n\n    Args:\n        model: The MLX model (must have model.layers or similar)\n        group: The distributed group\n        start_layer: First layer index for this rank (auto-computed if None)\n        end_layer: Last layer index (exclusive) for this rank (auto-computed if None)\n    \"\"\"\n    rank = group.rank()\n    world_size = group.size()\n\n    inner = get_inner_model(model)","sourceCodeStart":76,"sourceCodeEnd":112,"githubUrl":"https://github.com/mudler/LocalAI/blob/44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26/backend/python/mlx-distributed/sharding.py#L76-L112","documentation":"Raised by get_layers() in mlx-distributed/sharding.py when the inner model has neither a 'layers' nor an 'h' attribute. After locating the inner module, the pipeline-parallel code needs the layer list to compute each rank's slice, and these two names cover MLX/Llama-style stacks; anything else fails here.","triggerScenarios":"The inner model stores its transformer blocks under a different name (e.g. 'decoder.layers', 'blocks', 'layers_list') or is a Mamba/SSM model with no layered decoder at all, and pipeline_auto_parallel -> get_layers is called.","commonSituations":"Non-transformer or unusually structured architectures (Mamba, hybrid models), custom nn.Module containers, or mlx version drift renaming block collections.","solutions":["Inspect getattr(inner, attr) candidates and find the actual layer container","Add the attribute name to the ('layers', 'h') tuple in get_layers for your architecture","For non-layered architectures, pipeline parallelism does not apply — use another sharding strategy"],"exampleFix":"# before\nlayers = get_layers(inner)  # ValueError: must have 'layers' or 'h'\n\n# after\n# architecture stores blocks as 'blocks'\nlayers = inner.blocks\n# or patch: for attr in (\"layers\", \"h\", \"blocks\")","handlingStrategy":"type-guard","validationCode":"inner = get_inner_model(model)\nlayer_attrs = [a for a in ('layers', 'h') if getattr(inner, a, None) is not None]\nif not layer_attrs:\n    raise ValueError(f'{type(inner).__name__} exposes no layer list; pipeline sharding unsupported (attrs={dir(inner)})')","typeGuard":"def has_layer_list(inner) -> bool:\n    return any(getattr(inner, a, None) is not None for a in ('layers', 'h'))","tryCatchPattern":"try:\n    layers = get_layers(inner)\nexcept ValueError as err:\n    raise RuntimeError(f'pipeline parallelism unavailable for this model: {err}') from err","preventionTips":["Check for layers/h before enabling pipeline_auto_parallel","Non-layered (SSM/Mamba) architectures need different sharding","Extend the attribute tuple and upstream the change"],"tags":["python","mlx","sharding","model-structure"],"backgroundTag":null,"analyzedSha":"44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26","analyzedAt":"2026-08-15T10:13:50.291Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}