{"record":{"id":"ff9c51e3ae122a96","repo":"huggingface/transformers","slug":"fsdp-tp-is-not-supported-yet-use-distributedconfi","errorCode":null,"errorMessage":"FSDP+TP is not supported yet. Use DistributedConfig(fsdp_size=N) or DistributedConfig(tp_size=N), not both. 2D support will come soon.","messagePattern":"FSDP\\+TP is not supported yet\\. Use DistributedConfig\\(fsdp_size=N\\) or DistributedConfig\\(tp_size=N\\), not both\\. 2D support will come soon\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/distributed/configuration_utils.py","lineNumber":61,"sourceCode":"    tp_size: int | None = None\n    tp_plan: dict[str, str] | None = None\n    enable_sequence_parallel: bool = False\n    enable_expert_parallel: bool = False\n    fsdp_size: int | None = None\n    fsdp_cpu_offload: bool = False\n    fsdp_mixed_precision: bool = False\n\n    def __post_init__(self):\n        if self.tp_size is None and self.fsdp_size is None:\n            return\n\n        if self.tp_size is None:\n            self.tp_size = 1\n        if self.fsdp_size is None:\n            self.fsdp_size = 1\n\n        if self.tp_size > 1 and self.fsdp_size > 1:\n            raise ValueError(\n                \"FSDP+TP is not supported yet. \"\n                \"Use DistributedConfig(fsdp_size=N) or DistributedConfig(tp_size=N), not both. \"\n                \"2D support will come soon.\"\n            )\n\n    @classmethod\n    def from_dict(cls, config_dict: dict, **kwargs) -> \"DistributedConfig\":\n        merged = {**config_dict, **kwargs}\n        valid_keys = {f.name for f in cls.__dataclass_fields__.values()}\n        return cls(**{k: v for k, v in merged.items() if k in valid_keys})\n\n    def to_dict(self) -> dict:\n        return asdict(self)\n\n    def to_json_string(self) -> str:\n        return json.dumps(self.to_dict(), indent=2) + \"\\n\"\n\n    def to_json_file(self, json_file_path: str | os.PathLike):","sourceCodeStart":43,"sourceCodeEnd":79,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/distributed/configuration_utils.py#L43-L79","documentation":"DistributedConfig validates itself in __post_init__: tensor parallelism (tp_size>1) and FSDP (fsdp_size>1) cannot currently be combined in a single job - only pure TP or pure FSDP topologies are supported. Requesting both raises this ValueError immediately at config construction, before any distributed setup, with 2D (FSDPxTP) meshes planned for the future.","triggerScenarios":"Constructing DistributedConfig(tp_size=2, fsdp_size=2); loading a saved training config/distributed config JSON that contains both tp_size>1 and fsdp_size>1; migrating a multi-node setup where TP was added on top of an existing FSDP config.","commonSituations":"Trying to shard large models across fewer GPUs with TP while also wanting FSDP memory savings; config files copied from internal 2D-parallel experiments; tooling that merges distributed settings into one config.","solutions":["Pick one axis: DistributedConfig(tp_size=N, fsdp_size=None) or DistributedConfig(fsdp_size=N, tp_size=None).","If memory is the concern, prefer FSDP alone and/or increase CPU/NVMe offload rather than adding TP.","Check saved/merged config dicts for stray tp_size or fsdp_size keys before constructing DistributedConfig."],"exampleFix":"# before\ncfg = DistributedConfig(tp_size=2, fsdp_size=4)  # raises\n\n# after\ncfg = DistributedConfig(fsdp_size=8)  # pure FSDP\n# or\ncfg = DistributedConfig(tp_size=8)    # pure TP","handlingStrategy":"validation","validationCode":"def validate_distributed_config(tp_size, fsdp_size):\n    if (tp_size or 1) > 1 and (fsdp_size or 1) > 1:\n        raise ValueError(\"choose pure TP or pure FSDP; 2D meshes unsupported\")\n\nvalidate_distributed_config(cfg.get(\"tp_size\"), cfg.get(\"fsdp_size\"))","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Sanitize loaded config dicts: drop one of tp_size/fsdp_size before constructing DistributedConfig.","Document in your training scripts that only one parallel axis is supported.","Watch release notes for 2D FSDP+TP support before combining them."],"tags":["distributed","fsdp","tensor-parallelism","configuration"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}