huggingface/transformers · error · ValueError

Current shard-on-read only supports disjoint ranges on a sin

Error message

Current shard-on-read only supports disjoint ranges on a single checkpoint dimension.

What it means

During checkpoint loading with shard-on-read, _slice_and_cat reconstructs a full tensor from per-rank intervals. It supports the case where intervals are disjoint along ONE tensor dimension (multiple shards on a single checkpoint dimension); if more than one dimension has multiple intervals (e.g. both row-wise and column-wise StridedShard placements), it raises ValueError because no layout has been defined for that case yet.

Source

Thrown at src/transformers/distributed/sharding_utils.py:281

            if overlap_flat_start < overlap_flat_end:
                source_overlap_start = source_start + (overlap_flat_start - interval_flat_start)
                source_overlap_end = source_start + (overlap_flat_end - interval_flat_start)
                local_intervals.append((source_overlap_start, source_overlap_end))

        return local_intervals

    def _slice_and_cat(
        self,
        source: torch.Tensor,
        intervals: list[list[tuple[int, int]]],
        device: torch.device | str | int | None,
        dtype: torch.dtype | None,
    ) -> torch.Tensor:
        multi_interval_dims = [dim_idx for dim_idx, dim_intervals in enumerate(intervals) if len(dim_intervals) > 1]
        if len(multi_interval_dims) > 1:
            # NOTE(3outeille): not sure yet which scenario will have StridedShard
            # placements on both row and column. Thus, delay implementing this for now.
            raise ValueError("Current shard-on-read only supports disjoint ranges on a single checkpoint dimension.")
        concat_dim = multi_interval_dims[0] if multi_interval_dims else None

        base_slices = []
        for dim_idx, dim_intervals in enumerate(intervals):
            if dim_idx == concat_dim:
                # Disconnected intervals on this dim — placeholder; filled per interval below.
                base_slices.append(slice(None))
            else:
                # Single contiguous slice on this dim.
                start, end = dim_intervals[0]
                base_slices.append(slice(start, end))

        # Fast path: every dim is one contiguous interval, read in a single slice.
        if concat_dim is None:
            return source[tuple(base_slices)].to(device=device, dtype=dtype)

        # Multi-interval dim: keep base slices fixed and vary concat_dim only.
        base_slices_tuple = tuple(base_slices)

View on GitHub (pinned to a597f97485)

Solutions

1. Load the checkpoint without shard-on-read (full/gathered load) and let the library re-shard afterwards.
2. Adjust the tp_plan so each weight is sharded on a single dimension only (e.g. colwise OR rowwise, not both).
3. Re-save the checkpoint from a supported parallelization configuration so placements are 1-D.

Example fix

# before
plan = {"model.layers.*.self_attn.qkv_proj": "colwise", "model.layers.*.self_attn.qkv_proj": "rowwise"}  # unsupported 2-D sharding

# after
plan = {"model.layers.*.self_attn.q_proj": "colwise", "model.layers.*.mlp.up_proj": "colwise"}  # single-dim shards

def plan_is_single_dim(plan_intervals: dict[str, list[list[tuple[int, int]]]]) -> bool:
    return all(
        sum(1 for iv in intervals if len(iv) > 1) <= 1
        for intervals in plan_intervals.values()
    )

try:
    model = Model.from_pretrained(ckpt, distributed_config=cfg)
except ValueError as e:
    if "disjoint ranges on a single checkpoint dimension" in str(e):
        model = Model.from_pretrained(ckpt)  # full load, then re-shard manually
    else:
        raise

When it happens

Trigger: Loading a checkpoint whose parallel plan shards the same weight on two dimensions at once — e.g. a fused QKV or MLP weight that was saved with both dim0 and dim1 strided placements — through the shard-on-read path in sharding_utils.

Common situations: Combining TP plans that produce 2-D sharding on one layer; using custom StridedShard placements; loading old/experimental checkpoints produced outside the supported scheme.

Related errors

• {type(model).__name__} does not have a FSDP2 plan declared.
• Unsupported tensor parallel style '{parallel_style}' for lay
• tp_size ({distributed_config.tp_size}) * fsdp_size ({distrib
• FSDP+TP is not supported yet. Use DistributedConfig(fsdp_siz
• Expert parallelism was requested (`enable_expert_parallel=Tr

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/b90b88eeee7bdb34. Report an issue: GitHub.