{"record":{"id":"5e620ead230714b9","repo":"huggingface/transformers","slug":"unsupported-tensor-parallel-style-parallel-style","errorCode":null,"errorMessage":"Unsupported tensor parallel style '{parallel_style}' for layer '{layer_pattern}'. Supported styles are {list(ALL_PARALLEL_STYLES.keys())}","messagePattern":"Unsupported tensor parallel style '(.+?)' for layer '(.+?)'\\. Supported styles are (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/distributed/mixin.py","lineNumber":119,"sourceCode":"    @property\n    def fsdp_plan(self) -> dict[str, str]:\n        return self._fsdp_plan\n\n    @property\n    def pp_plan(self) -> dict[str, tuple[str, str]]:\n        return self._pp_plan\n\n    @tp_plan.setter\n    def tp_plan(self, plan: dict[str, str] | None):\n        if plan is None:\n            self._tp_plan = {}\n            return\n        if not isinstance(plan, dict):\n            raise ValueError(\"Can only set a dictionary as `tp_plan`\")\n\n        for layer_pattern, parallel_style in plan.items():\n            if parallel_style not in ALL_PARALLEL_STYLES:\n                raise ValueError(\n                    f\"Unsupported tensor parallel style '{parallel_style}' for layer '{layer_pattern}'. \"\n                    f\"Supported styles are {list(ALL_PARALLEL_STYLES.keys())}\"\n                )\n\n        model_param_names = [name for name, _ in self.named_parameters()]\n        for layer_pattern in plan.keys():\n            regex_pattern = layer_pattern.replace(\"*\", r\"\\d+\")\n            pattern_matched = False\n            for param_name in model_param_names:\n                if re.match(regex_pattern, param_name):\n                    pattern_matched = True\n                    break\n            if not pattern_matched:\n                warnings.warn(\n                    f\"Layer pattern '{layer_pattern}' does not match any parameters in the model. This rule may not \"\n                    \"be applied during tensor parallelization, or may lead to dimension mismatches\"\n                )\n","sourceCodeStart":101,"sourceCodeEnd":137,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/distributed/mixin.py#L101-L137","documentation":"The tp_plan setter on a model with the distributed mixin validates every value of the plan dictionary against the registry ALL_PARALLEL_STYLES. If a layer pattern is mapped to a style name that is not registered (e.g. a typo or an unsupported parallelism mode), a ValueError is raised before any sharding happens. The message lists the exact set of supported style keys so the correction is mechanical.","triggerScenarios":"Calling model.tp_plan = {...} (or passing distributed_config={'tp_plan': {...}} to from_pretrained) with a dict whose value string is not a key of ALL_PARALLEL_STYLES, e.g. {'model.layers.*.self_attn': 'column-wise'} or 'colwise_parallel' instead of 'colwise'.","commonSituations":"Typos in style names ('col-wise' vs 'colwise'), copying plan examples from older blog posts/docs that use different names, or assuming a style exists for a model that does not support it.","solutions":["Fix the style name to one of the keys printed in the error message (typical values: 'colwise', 'rowwise', 'colwise_rep', 'rowwise_rep', 'standard').","If unsure which layers to parallelize, pass tp_plan='auto' instead of a dict and let the library choose.","Remove the offending layer pattern from the plan if that layer should stay replicated."],"exampleFix":"# before\nconfig.distributed_config = {\"tp_plan\": {\"model.layers.*.self_attn\": \"column-parallel\"}, \"tp_size\": 2}\n\n# after\nconfig.distributed_config = {\"tp_plan\": {\"model.layers.*.self_attn\": \"colwise\"}, \"tp_size\": 2}","handlingStrategy":"validation","validationCode":"from transformers.distributed.tensor_parallel import ALL_PARALLEL_STYLES  # module path may vary; else:\n# from transformers.distributed import ALL_PARALLEL_STYLES\n\ndef validate_tp_plan(plan: dict) -> None:\n    bad = {k: v for k, v in plan.items() if v not in ALL_PARALLEL_STYLES}\n    if bad:\n        raise ValueError(f\"Invalid TP styles {bad}; valid: {sorted(ALL_PARALLEL_STYLES)}\")","typeGuard":"def is_valid_tp_plan(plan: object) -> bool:\n    return isinstance(plan, dict) and all(v in ALL_PARALLEL_STYLES for v in plan.values())","tryCatchPattern":"try:\n    model.tp_plan = plan\nexcept ValueError as e:\n    if \"Unsupported tensor parallel style\" in str(e):\n        logger.error(\"bad tp_plan style; falling back to auto\")\n        model.tp_plan = \"auto\"\n    else:\n        raise","preventionTips":["Keep tp_plan dicts in one config module and validate them at startup against ALL_PARALLEL_STYLES.","Prefer tp_plan='auto' unless you know the layer layout.","Add a unit test asserting every style in your plan is a key of ALL_PARALLEL_STYLES."],"tags":["tensor-parallel","distributed","config","validation"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}