huggingface/transformers · error · ValueError

Unsupported tensor parallel style

Error message

Unsupported tensor parallel style '{parallel_style}' for layer '{layer_pattern}'. Supported styles are {list(ALL_PARALLEL_STYLES.keys())}

What it means

The tp_plan setter on a model with the distributed mixin validates every value of the plan dictionary against the registry ALL_PARALLEL_STYLES. If a layer pattern is mapped to a style name that is not registered (e.g. a typo or an unsupported parallelism mode), a ValueError is raised before any sharding happens. The message lists the exact set of supported style keys so the correction is mechanical.

Solutions

Fix the style name to one of the keys printed in the error message (typical values: 'colwise', 'rowwise', 'colwise_rep', 'rowwise_rep', 'standard').
If unsure which layers to parallelize, pass tp_plan='auto' instead of a dict and let the library choose.
Remove the offending layer pattern from the plan if that layer should stay replicated.

Example fix

# before
config.distributed_config = {"tp_plan": {"model.layers.*.self_attn": "column-parallel"}, "tp_size": 2}

# after
config.distributed_config = {"tp_plan": {"model.layers.*.self_attn": "colwise"}, "tp_size": 2}

Defensive patterns

Strategy: validation

Validate before calling

from transformers.distributed.tensor_parallel import ALL_PARALLEL_STYLES  # module path may vary; else:
# from transformers.distributed import ALL_PARALLEL_STYLES

def validate_tp_plan(plan: dict) -> None:
    bad = {k: v for k, v in plan.items() if v not in ALL_PARALLEL_STYLES}
    if bad:
        raise ValueError(f"Invalid TP styles {bad}; valid: {sorted(ALL_PARALLEL_STYLES)}")

Type guard

def is_valid_tp_plan(plan: object) -> bool:
    return isinstance(plan, dict) and all(v in ALL_PARALLEL_STYLES for v in plan.values())

Try / catch

try:
    model.tp_plan = plan
except ValueError as e:
    if "Unsupported tensor parallel style" in str(e):
        logger.error("bad tp_plan style; falling back to auto")
        model.tp_plan = "auto"
    else:
        raise

Prevention

Keep tp_plan dicts in one config module and validate them at startup against ALL_PARALLEL_STYLES.
Prefer tp_plan='auto' unless you know the layer layout.
Add a unit test asserting every style in your plan is a key of ALL_PARALLEL_STYLES.

When it happens

Trigger: Calling model.tp_plan = {...} (or passing distributed_config={'tp_plan': {...}} to from_pretrained) with a dict whose value string is not a key of ALL_PARALLEL_STYLES, e.g. {'model.layers.*.self_attn': 'column-wise'} or 'colwise_parallel' instead of 'colwise'.

Common situations: Typos in style names ('col-wise' vs 'colwise'), copying plan examples from older blog posts/docs that use different names, or assuming a style exists for a model that does not support it.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/5e620ead230714b9. Report an issue: GitHub.

Appendix: source

Thrown at src/transformers/distributed/mixin.py:119

    @property
    def fsdp_plan(self) -> dict[str, str]:
        return self._fsdp_plan

    @property
    def pp_plan(self) -> dict[str, tuple[str, str]]:
        return self._pp_plan

    @tp_plan.setter
    def tp_plan(self, plan: dict[str, str] | None):
        if plan is None:
            self._tp_plan = {}
            return
        if not isinstance(plan, dict):
            raise ValueError("Can only set a dictionary as `tp_plan`")

        for layer_pattern, parallel_style in plan.items():
            if parallel_style not in ALL_PARALLEL_STYLES:
                raise ValueError(
                    f"Unsupported tensor parallel style '{parallel_style}' for layer '{layer_pattern}'. "
                    f"Supported styles are {list(ALL_PARALLEL_STYLES.keys())}"
                )

        model_param_names = [name for name, _ in self.named_parameters()]
        for layer_pattern in plan.keys():
            regex_pattern = layer_pattern.replace("*", r"\d+")
            pattern_matched = False
            for param_name in model_param_names:
                if re.match(regex_pattern, param_name):
                    pattern_matched = True
                    break
            if not pattern_matched:
                warnings.warn(
                    f"Layer pattern '{layer_pattern}' does not match any parameters in the model. This rule may not "
                    "be applied during tensor parallelization, or may lead to dimension mismatches"
                )

View on GitHub (pinned to a597f97485)