huggingface/transformers · error · ValueError

FSDP+TP is not supported yet. Use DistributedConfig(fsdp_siz

Error message

FSDP+TP is not supported yet. Use DistributedConfig(fsdp_size=N) or DistributedConfig(tp_size=N), not both. 2D support will come soon.

What it means

DistributedConfig validates itself in __post_init__: tensor parallelism (tp_size>1) and FSDP (fsdp_size>1) cannot currently be combined in a single job - only pure TP or pure FSDP topologies are supported. Requesting both raises this ValueError immediately at config construction, before any distributed setup, with 2D (FSDPxTP) meshes planned for the future.

Source

Thrown at src/transformers/distributed/configuration_utils.py:61

    tp_size: int | None = None
    tp_plan: dict[str, str] | None = None
    enable_sequence_parallel: bool = False
    enable_expert_parallel: bool = False
    fsdp_size: int | None = None
    fsdp_cpu_offload: bool = False
    fsdp_mixed_precision: bool = False

    def __post_init__(self):
        if self.tp_size is None and self.fsdp_size is None:
            return

        if self.tp_size is None:
            self.tp_size = 1
        if self.fsdp_size is None:
            self.fsdp_size = 1

        if self.tp_size > 1 and self.fsdp_size > 1:
            raise ValueError(
                "FSDP+TP is not supported yet. "
                "Use DistributedConfig(fsdp_size=N) or DistributedConfig(tp_size=N), not both. "
                "2D support will come soon."
            )

    @classmethod
    def from_dict(cls, config_dict: dict, **kwargs) -> "DistributedConfig":
        merged = {**config_dict, **kwargs}
        valid_keys = {f.name for f in cls.__dataclass_fields__.values()}
        return cls(**{k: v for k, v in merged.items() if k in valid_keys})

    def to_dict(self) -> dict:
        return asdict(self)

    def to_json_string(self) -> str:
        return json.dumps(self.to_dict(), indent=2) + "\n"

    def to_json_file(self, json_file_path: str | os.PathLike):

View on GitHub (pinned to a597f97485)

Solutions

1. Pick one axis: DistributedConfig(tp_size=N, fsdp_size=None) or DistributedConfig(fsdp_size=N, tp_size=None).
2. If memory is the concern, prefer FSDP alone and/or increase CPU/NVMe offload rather than adding TP.
3. Check saved/merged config dicts for stray tp_size or fsdp_size keys before constructing DistributedConfig.

Example fix

# before
cfg = DistributedConfig(tp_size=2, fsdp_size=4)  # raises

# after
cfg = DistributedConfig(fsdp_size=8)  # pure FSDP
# or
cfg = DistributedConfig(tp_size=8)    # pure TP

def validate_distributed_config(tp_size, fsdp_size):
    if (tp_size or 1) > 1 and (fsdp_size or 1) > 1:
        raise ValueError("choose pure TP or pure FSDP; 2D meshes unsupported")

validate_distributed_config(cfg.get("tp_size"), cfg.get("fsdp_size"))

When it happens

Trigger: Constructing DistributedConfig(tp_size=2, fsdp_size=2); loading a saved training config/distributed config JSON that contains both tp_size>1 and fsdp_size>1; migrating a multi-node setup where TP was added on top of an existing FSDP config.

Common situations: Trying to shard large models across fewer GPUs with TP while also wanting FSDP memory savings; config files copied from internal 2D-parallel experiments; tooling that merges distributed settings into one config.

Related errors

• Expert parallelism was requested (`enable_expert_parallel=Tr
• Can only set a dictionary as `tp_plan`
• tp_size ({distributed_config.tp_size}) * fsdp_size ({distrib
• save_pretrained(..., distributed_checkpoint=True) is only su
• save_pretrained(..., distributed_checkpoint=True) requires t

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/ff9c51e3ae122a96. Report an issue: GitHub.