{"record":{"id":"52e7a3829d206be9","repo":"Lightning-AI/pytorch-lightning","slug":"failed-to-load-checkpoint-directly-into-the-model","errorCode":null,"errorMessage":"Failed to load checkpoint directly into the model. The given path must be a single file containing the full state dict: {path}","messagePattern":"Failed to load checkpoint directly into the model\\. The given path must be a single file containing the full state dict: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/strategies/model_parallel.py","lineNumber":547,"sourceCode":"            f\" ({world_size}).\"\n        )\n    return init_device_mesh(\n        device_type=device.type,\n        mesh_shape=(data_parallel_size, tensor_parallel_size),\n        mesh_dim_names=(\"data_parallel\", \"tensor_parallel\"),\n    )\n\n\ndef _has_dtensor_modules(module: object) -> TypeGuard[Module]:\n    from torch.distributed._tensor import DTensor\n\n    return isinstance(module, Module) and any(isinstance(t, DTensor) for t in module.parameters())\n\n\ndef _load_raw_module_state_from_path(path: _PATH, module: Module, world_size: int, strict: bool = True) -> None:\n    \"\"\"Loads the state dict from a file path into the FSDP module.\"\"\"\n    if not _is_full_checkpoint(path):\n        raise ValueError(\n            \"Failed to load checkpoint directly into the model. The given path must be a single file containing the\"\n            f\" full state dict: {path}\"\n        )\n    if _is_local_file_protocol(str(path)):\n        # Use `mmap` to avoid storing a copy of the full checkpoint per rank\n        state_dict = torch.load(path, mmap=True, map_location=\"cpu\")\n    else:\n        state_dict = _load(path, map_location=\"cpu\")\n    _load_raw_module_state(state_dict=state_dict, module=module, world_size=world_size, strict=strict)\n\n\ndef _load_raw_module_state(\n    state_dict: dict[str, Any], module: Module, world_size: int = 1, strict: bool = True\n) -> None:\n    \"\"\"Loads the state dict into the module by gathering all weights first and then and writing back to each shard.\"\"\"\n    from torch.distributed.fsdp import FullyShardedDataParallel as FSDP\n\n    if _has_dtensor_modules(module):","sourceCodeStart":529,"sourceCodeEnd":565,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/strategies/model_parallel.py#L529-L565","documentation":"_load_raw_module_state_from_path loads a checkpoint directly into a module and requires the path to be a single file containing the full state dict (a 'full checkpoint'), not a directory of shards. _is_full_checkpoint returned False, so the load is rejected before torch.load.","triggerScenarios":"Calling the raw load path (e.g. load_checkpoint with no distributed model in state, which delegates to _load_raw_module_state_from_path) with a path that is a directory of distributed shards or an otherwise non-file/non-checkpoint location.","commonSituations":"Mixing checkpoint formats: a sharded checkpoint saved by ModelParallelStrategy later loaded via a path/code path expecting one consolidated file; remote URLs where the file-existence/full-checkpoint probe fails; resuming from a .distcp directory.","solutions":["Consolidate the sharded checkpoint into a single file first (e.g. torch.distributed.checkpoint.format_utils.dcp_to_torch_save) and point at that file","Or load through the distributed path: pass state={'model': model} with the module set up via the strategy so _load_checkpoint handles the shard directory","Verify the path is a regular file containing a state dict"],"exampleFix":"# before\nstrategy.load_checkpoint('ckpt_dir', state={'other': metadata})\n# after (consolidate shards into one file)\n# python -m torch.distributed.checkpoint.format_utils --dcp_to_torch_save ckpt_dir consolidated.pt\nstrategy.load_checkpoint('consolidated.pt', state={'other': metadata})","handlingStrategy":"validation","validationCode":"p = Path(path)\nassert p.is_file(), f'{path} must be a single full-checkpoint file, not a shard directory'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep track of whether a checkpoint is sharded or consolidated","Consolidate shard dirs with dcp_to_torch_save before raw loading"],"tags":["lightning","fabric","checkpoint","full-checkpoint","shards"],"backgroundTag":"checkpoint-format-mismatch","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}