{"record":{"id":"2478842a15eee665","repo":"mudler/LocalAI","slug":"model-must-have-a-model-or-transformer-attribu","errorCode":null,"errorMessage":"Model must have a 'model' or 'transformer' attribute","messagePattern":"Model must have a 'model' or 'transformer' attribute","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"backend/python/mlx-distributed/sharding.py","lineNumber":85,"sourceCode":"        # Gather output from all ranks so every rank has the final result\n        output = mx.distributed.all_gather(output, group=self.group)[\n            -output.shape[0] :\n        ]\n        mx.eval(output)\n        return output\n\n\ndef get_inner_model(model):\n    \"\"\"Get the inner model (model.model or model.transformer).\"\"\"\n    for attr in (\"model\", \"transformer\"):\n        inner = getattr(model, attr, None)\n        if isinstance(inner, nn.Module):\n            # Some models have model.model (e.g. language_model.model)\n            inner_inner = getattr(inner, \"model\", None)\n            if isinstance(inner_inner, nn.Module):\n                return inner_inner\n            return inner\n    raise ValueError(\"Model must have a 'model' or 'transformer' attribute\")\n\n\ndef get_layers(inner_model):\n    \"\"\"Get the list of transformer layers.\"\"\"\n    for attr in (\"layers\", \"h\"):\n        layers = getattr(inner_model, attr, None)\n        if layers is not None:\n            return layers\n    raise ValueError(\"Model must have a 'layers' or 'h' attribute\")\n\n\ndef pipeline_auto_parallel(model, group, start_layer=None, end_layer=None):\n    \"\"\"Apply pipeline parallelism to a model.\n\n    Each rank only keeps its slice of layers.  The first layer receives from\n    the previous rank, and the last layer sends to the next rank.\n\n    Args:","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/mudler/LocalAI/blob/44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26/backend/python/mlx-distributed/sharding.py#L67-L103","documentation":"Raised by get_inner_model() in mlx-distributed/sharding.py when the wrapped model exposes neither a 'model' nor a 'transformer' attribute that is an nn.Module. The sharding code needs the inner transformer stack to slice layers across ranks, so it walks two common attribute names and fails if neither matches the model's structure.","triggerScenarios":"Loading a model class that nests its layers differently (e.g. attribute named 'language_model', 'backbone', or layers directly on the top-level object) and then calling pipeline_auto_parallel/get_inner_model on it.","commonSituations":"New or unusual HF-compatible MLX model architectures whose internals don't follow the model.model/model.transformer convention; mlx-lm version changes renaming attributes; custom model wrappers.","solutions":["Inspect the model with vars(model)/dir(model) to find the real attribute holding the transformer stack","Wrap or alias: model.model = <inner module> before sharding","Extend the ('model', 'transformer') tuple in get_inner_model with the attribute your architecture uses (and upstream it)"],"exampleFix":"# before\ninner = get_inner_model(model)  # ValueError: must have 'model' or 'transformer'\n\n# after\n# for models exposing 'language_model'\nmodel.model = model.language_model\ninner = get_inner_model(model)","handlingStrategy":"type-guard","validationCode":"import mlx.nn as nn\ncandidates = [getattr(model, a, None) for a in ('model', 'transformer')]\nif not any(isinstance(c, nn.Module) for c in candidates):\n    raise ValueError(f'unsupported architecture {type(model).__name__}: cannot locate inner transformer; attrs={ [k for k,v in vars(model).items() if isinstance(v, nn.Module)] }')","typeGuard":"def has_inner_module(model) -> bool:\n    return any(isinstance(getattr(model, a, None), nn.Module) for a in ('model', 'transformer'))","tryCatchPattern":"try:\n    inner = get_inner_model(model)\nexcept ValueError as err:\n    raise RuntimeError(f'cannot shard {type(model).__name__}: {err}') from err","preventionTips":["Smoke-test sharding on new model architectures before deploying","Log the model's nn.Module attributes at load for diagnosability","Keep a whitelist of verified architectures per sharding path"],"tags":["python","mlx","sharding","model-structure"],"backgroundTag":null,"analyzedSha":"44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26","analyzedAt":"2026-08-15T10:13:50.291Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}