huggingface/transformers · error · ValueError

Unsupported tensor parallel style '{parallel_style}' for lay

Error message

Unsupported tensor parallel style '{parallel_style}' for layer '{layer_pattern}'. Supported styles are {list(ALL_PARALLEL_STYLES.keys())}

What it means

The tp_plan setter on a model with the distributed mixin validates every value of the plan dictionary against the registry ALL_PARALLEL_STYLES. If a layer pattern is mapped to a style name that is not registered (e.g. a typo or an unsupported parallelism mode), a ValueError is raised before any sharding happens. The message lists the exact set of supported style keys so the correction is mechanical.

Source

Thrown at src/transformers/distributed/mixin.py:119

    @property
    def fsdp_plan(self) -> dict[str, str]:
        return self._fsdp_plan

    @property
    def pp_plan(self) -> dict[str, tuple[str, str]]:
        return self._pp_plan

    @tp_plan.setter
    def tp_plan(self, plan: dict[str, str] | None):
        if plan is None:
            self._tp_plan = {}
            return
        if not isinstance(plan, dict):
            raise ValueError("Can only set a dictionary as `tp_plan`")

        for layer_pattern, parallel_style in plan.items():
            if parallel_style not in ALL_PARALLEL_STYLES:
                raise ValueError(
                    f"Unsupported tensor parallel style '{parallel_style}' for layer '{layer_pattern}'. "
                    f"Supported styles are {list(ALL_PARALLEL_STYLES.keys())}"
                )

        model_param_names = [name for name, _ in self.named_parameters()]
        for layer_pattern in plan.keys():
            regex_pattern = layer_pattern.replace("*", r"\d+")
            pattern_matched = False
            for param_name in model_param_names:
                if re.match(regex_pattern, param_name):
                    pattern_matched = True
                    break
            if not pattern_matched:
                warnings.warn(
                    f"Layer pattern '{layer_pattern}' does not match any parameters in the model. This rule may not "
                    "be applied during tensor parallelization, or may lead to dimension mismatches"
                )

View on GitHub (pinned to a597f97485)

Solutions

Fix the style name to one of the keys printed in the error message (typical values: 'colwise', 'rowwise', 'colwise_rep', 'rowwise_rep', 'standard').
If unsure which layers to parallelize, pass tp_plan='auto' instead of a dict and let the library choose.
Remove the offending layer pattern from the plan if that layer should stay replicated.

Example fix

# before
config.distributed_config = {"tp_plan": {"model.layers.*.self_attn": "column-parallel"}, "tp_size": 2}

# after
config.distributed_config = {"tp_plan": {"model.layers.*.self_attn": "colwise"}, "tp_size": 2}

Defensive patterns

Strategy: validation

Validate before calling

from transformers.distributed.tensor_parallel import ALL_PARALLEL_STYLES  # module path may vary; else:
# from transformers.distributed import ALL_PARALLEL_STYLES

def validate_tp_plan(plan: dict) -> None:
    bad = {k: v for k, v in plan.items() if v not in ALL_PARALLEL_STYLES}
    if bad:
        raise ValueError(f"Invalid TP styles {bad}; valid: {sorted(ALL_PARALLEL_STYLES)}")

Type guard

def is_valid_tp_plan(plan: object) -> bool:
    return isinstance(plan, dict) and all(v in ALL_PARALLEL_STYLES for v in plan.values())

Try / catch

try:
    model.tp_plan = plan
except ValueError as e:
    if "Unsupported tensor parallel style" in str(e):
        logger.error("bad tp_plan style; falling back to auto")
        model.tp_plan = "auto"
    else:
        raise

Prevention

Keep tp_plan dicts in one config module and validate them at startup against ALL_PARALLEL_STYLES.
Prefer tp_plan='auto' unless you know the layer layout.
Add a unit test asserting every style in your plan is a key of ALL_PARALLEL_STYLES.

When it happens

Trigger: Calling model.tp_plan = {...} (or passing distributed_config={'tp_plan': {...}} to from_pretrained) with a dict whose value string is not a key of ALL_PARALLEL_STYLES, e.g. {'model.layers.*.self_attn': 'column-wise'} or 'colwise_parallel' instead of 'colwise'.

Common situations: Typos in style names ('col-wise' vs 'colwise'), copying plan examples from older blog posts/docs that use different names, or assuming a style exists for a model that does not support it.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/5e620ead230714b9. Report an issue: GitHub.