{"record":{"id":"b90b88eeee7bdb34","repo":"huggingface/transformers","slug":"current-shard-on-read-only-supports-disjoint-range","errorCode":null,"errorMessage":"Current shard-on-read only supports disjoint ranges on a single checkpoint dimension.","messagePattern":"Current shard-on-read only supports disjoint ranges on a single checkpoint dimension\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/distributed/sharding_utils.py","lineNumber":281,"sourceCode":"            if overlap_flat_start < overlap_flat_end:\n                source_overlap_start = source_start + (overlap_flat_start - interval_flat_start)\n                source_overlap_end = source_start + (overlap_flat_end - interval_flat_start)\n                local_intervals.append((source_overlap_start, source_overlap_end))\n\n        return local_intervals\n\n    def _slice_and_cat(\n        self,\n        source: torch.Tensor,\n        intervals: list[list[tuple[int, int]]],\n        device: torch.device | str | int | None,\n        dtype: torch.dtype | None,\n    ) -> torch.Tensor:\n        multi_interval_dims = [dim_idx for dim_idx, dim_intervals in enumerate(intervals) if len(dim_intervals) > 1]\n        if len(multi_interval_dims) > 1:\n            # NOTE(3outeille): not sure yet which scenario will have StridedShard\n            # placements on both row and column. Thus, delay implementing this for now.\n            raise ValueError(\"Current shard-on-read only supports disjoint ranges on a single checkpoint dimension.\")\n        concat_dim = multi_interval_dims[0] if multi_interval_dims else None\n\n        base_slices = []\n        for dim_idx, dim_intervals in enumerate(intervals):\n            if dim_idx == concat_dim:\n                # Disconnected intervals on this dim — placeholder; filled per interval below.\n                base_slices.append(slice(None))\n            else:\n                # Single contiguous slice on this dim.\n                start, end = dim_intervals[0]\n                base_slices.append(slice(start, end))\n\n        # Fast path: every dim is one contiguous interval, read in a single slice.\n        if concat_dim is None:\n            return source[tuple(base_slices)].to(device=device, dtype=dtype)\n\n        # Multi-interval dim: keep base slices fixed and vary concat_dim only.\n        base_slices_tuple = tuple(base_slices)","sourceCodeStart":263,"sourceCodeEnd":299,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/distributed/sharding_utils.py#L263-L299","documentation":"During checkpoint loading with shard-on-read, _slice_and_cat reconstructs a full tensor from per-rank intervals. It supports the case where intervals are disjoint along ONE tensor dimension (multiple shards on a single checkpoint dimension); if more than one dimension has multiple intervals (e.g. both row-wise and column-wise StridedShard placements), it raises ValueError because no layout has been defined for that case yet.","triggerScenarios":"Loading a checkpoint whose parallel plan shards the same weight on two dimensions at once — e.g. a fused QKV or MLP weight that was saved with both dim0 and dim1 strided placements — through the shard-on-read path in sharding_utils.","commonSituations":"Combining TP plans that produce 2-D sharding on one layer; using custom StridedShard placements; loading old/experimental checkpoints produced outside the supported scheme.","solutions":["Load the checkpoint without shard-on-read (full/gathered load) and let the library re-shard afterwards.","Adjust the tp_plan so each weight is sharded on a single dimension only (e.g. colwise OR rowwise, not both).","Re-save the checkpoint from a supported parallelization configuration so placements are 1-D."],"exampleFix":"# before\nplan = {\"model.layers.*.self_attn.qkv_proj\": \"colwise\", \"model.layers.*.self_attn.qkv_proj\": \"rowwise\"}  # unsupported 2-D sharding\n\n# after\nplan = {\"model.layers.*.self_attn.q_proj\": \"colwise\", \"model.layers.*.mlp.up_proj\": \"colwise\"}  # single-dim shards","handlingStrategy":"fallback","validationCode":"def plan_is_single_dim(plan_intervals: dict[str, list[list[tuple[int, int]]]]) -> bool:\n    return all(\n        sum(1 for iv in intervals if len(iv) > 1) <= 1\n        for intervals in plan_intervals.values()\n    )","typeGuard":null,"tryCatchPattern":"try:\n    model = Model.from_pretrained(ckpt, distributed_config=cfg)\nexcept ValueError as e:\n    if \"disjoint ranges on a single checkpoint dimension\" in str(e):\n        model = Model.from_pretrained(ckpt)  # full load, then re-shard manually\n    else:\n        raise","preventionTips":["Keep each weight sharded on one dimension in tp_plan.","Re-emit checkpoints from supported parallelization schemes.","When unsure, load without shard-on-read and re-shard after load."],"tags":["tensor-parallel","sharding","checkpoint-loading","distributed"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}