{"record":{"id":"8420d43eee2c4462","repo":"Lightning-AI/pytorch-lightning","slug":"the-sizes-data-parallel-size-data-parallel-size","errorCode":null,"errorMessage":"The sizes `data_parallel_size={data_parallel_size}` and `tensor_parallel_size={tensor_parallel_size}` multiplied should equal the world size ({world_size}).","messagePattern":"The sizes `data_parallel_size=(.+?)` and `tensor_parallel_size=(.+?)` multiplied should equal the world size \\((.+?)\\)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/strategies/model_parallel.py","lineNumber":526,"sourceCode":"        # return the remaining metadata that wasn't requested as part of `state`\n        return checkpoint\n\n    raise ValueError(\n        f\"The path {str(path)!r} does not point to a valid checkpoint. Make sure the path points to either a\"\n        \" directory with distributed checkpoint shards, or a single file with a full checkpoint.\"\n    )\n\n\ndef _setup_device_mesh(\n    data_parallel_size: int,\n    tensor_parallel_size: int,\n    world_size: int,\n    device: torch.device,\n) -> \"DeviceMesh\":\n    from torch.distributed.device_mesh import init_device_mesh\n\n    if data_parallel_size * tensor_parallel_size != world_size:\n        raise RuntimeError(\n            f\"The sizes `data_parallel_size={data_parallel_size}` and\"\n            f\" `tensor_parallel_size={tensor_parallel_size}` multiplied should equal the world size\"\n            f\" ({world_size}).\"\n        )\n    return init_device_mesh(\n        device_type=device.type,\n        mesh_shape=(data_parallel_size, tensor_parallel_size),\n        mesh_dim_names=(\"data_parallel\", \"tensor_parallel\"),\n    )\n\n\ndef _has_dtensor_modules(module: object) -> TypeGuard[Module]:\n    from torch.distributed._tensor import DTensor\n\n    return isinstance(module, Module) and any(isinstance(t, DTensor) for t in module.parameters())\n\n\ndef _load_raw_module_state_from_path(path: _PATH, module: Module, world_size: int, strict: bool = True) -> None:","sourceCodeStart":508,"sourceCodeEnd":544,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/strategies/model_parallel.py#L508-L544","documentation":"_setup_device_mesh validates that data_parallel_size * tensor_parallel_size equals the distributed world size before calling torch.distributed.device_mesh.init_device_mesh. A mismatch means the parallel layout cannot map onto the available ranks.","triggerScenarios":"Creating ModelParallelStrategy(data_parallel_size=2, tensor_parallel_size=2) and launching with world_size != 4 (e.g. 2 GPUs, or 8 GPUs); using a different number of processes/devices than the strategy's parallel sizes imply.","commonSituations":"Scaling a TP config from one machine size to another (e.g. 8xGPU config run on 2xGPU) without adjusting sizes; env var overrides of world size; mixed use of devices=['auto'] and hardcoded sizes.","solutions":["Set data_parallel_size * tensor_parallel_size == world_size (e.g. world_size=4 → dp=2, tp=2, or dp=1, tp=4)","Or change the launcher/process count to match the configured sizes (e.g. torchrun --nproc_per_node=4)","Compute sizes dynamically: tensor_parallel_size = min(world_size, desired_tp); data_parallel_size = world_size // tensor_parallel_size"],"exampleFix":"# before\nstrategy = ModelParallelStrategy(tensor_parallel_size=4)  # run with world_size=2\n# after\nworld_size = torch.distributed.get_world_size()\ntp = min(4, world_size)\nstrategy = ModelParallelStrategy(tensor_parallel_size=tp, data_parallel_size=world_size // tp)","handlingStrategy":"validation","validationCode":"ws = torch.distributed.get_world_size()\nassert data_parallel_size * tensor_parallel_size == ws, (\n    f'dp({data_parallel_size}) * tp({tensor_parallel_size}) != world_size({ws})'\n)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Derive dp/tp sizes from world_size instead of hardcoding","Parameterize sizes via config tied to the launcher's nproc_per_node"],"tags":["lightning","fabric","model-parallel","device-mesh","world-size"],"backgroundTag":"device-mesh-size-mismatch","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}