sgl-project/sglang · error · TypeError

Rank-local FSDP shard produced for non-DTensor parameter {ta

Error message

Rank-local FSDP shard produced for non-DTensor parameter {target_param_name}

What it means

Raised when the FSDP loader takes the rank-local FSDP sharding path for a parameter whose meta-model counterpart is not a torch.distributed.tensor.DTensor. The rank-local FSDP branch assumes the target parameter is FSDP-sharded (a DTensor) so it can derive the local shard; a plain tensor indicates the parameter was never registered for FSDP sharding.

Source

Thrown at python/sglang/multimodal_gen/runtime/loader/fsdp_load.py:677

                    len(quantized_dtype_mismatch_examples[mismatch_key])
                    < _DTYPE_MISMATCH_EXAMPLE_LIMIT
                ):
                    quantized_dtype_mismatch_examples[mismatch_key].append(
                        target_param_name
                    )
            else:
                non_quantized_dtype_mismatch_counts[mismatch_key] += 1
                if (
                    len(non_quantized_dtype_mismatch_examples[mismatch_key])
                    < _DTYPE_MISMATCH_EXAMPLE_LIMIT
                ):
                    non_quantized_dtype_mismatch_examples[mismatch_key].append(
                        target_param_name
                    )

        if is_rank_local_fsdp_shard:
            if not isinstance(meta_sharded_param, dist_tensor.DTensor):
                raise TypeError(
                    f"Rank-local FSDP shard produced for non-DTensor parameter {target_param_name}"
                )
            local_tensor = full_tensor.to(
                device=checkpoint_load_device,
                dtype=target_dtype,
            )
            sharded_tensor = dist_tensor.DTensor.from_local(
                local_tensor,
                meta_sharded_param.device_mesh,
                meta_sharded_param.placements,
                run_check=False,
                shape=meta_sharded_param.shape,
                stride=meta_sharded_param.stride(),
            )
            if cpu_offload:
                sharded_tensor = sharded_tensor.to("cpu")
        elif is_rank_local_tp_shard:
            if isinstance(meta_sharded_param, dist_tensor.DTensor):

View on GitHub (pinned to 0132848349)

Solutions

1. Check how is_rank_local_fsdp_shard was computed for this parameter and why meta_sharded_param is not a DTensor (print type(model parameter) on meta init)
2. Fix the FSDP wrap policy / sharding metadata so the parameter is actually FSDP-sharded (DTensor) or routed to the plain-tensor path
3. Ensure the model is meta-initialized under the same FSDP configuration used at load time

Example fix

# before: param excluded from FSDP wrap, loader treats it as rank-local shard
# after: include the module in the FSDP wrap policy so its params become DTensors
auto_wrap_policy = functools.partial(
    size_based_auto_wrap_policy, min_num_params=int(1e6)
)  # ensure the module holding target_param_name is wrapped

import torch.distributed.tensor as dist_tensor
for name, p in model.named_parameters():
    if sharding_plan[name].is_rank_local_fsdp_shard:
        assert isinstance(p, dist_tensor.DTensor), f"{name} not FSDP-sharded"

def is_fsdp_dtensor(p) -> bool:
    import torch.distributed.tensor as dist_tensor
    return isinstance(p, dist_tensor.DTensor)

try:
    load_model_from_full_model_state_dict(...)
except TypeError as e:
    if 'Rank-local FSDP shard' in str(e):
        fix_wrap_policy(); retry_load()

When it happens

Trigger: Calling load_model_from_full_model_state_dict (directly or via maybe_load_fsdp_model / _load_weights_into_model / _load_dit_model) where the sharding metadata flags a parameter as is_rank_local_fsdp_shard=True but model.meta_parameters[name] is a plain torch.Tensor rather than a DTensor — e.g. a module wrapped incorrectly or excluded from the FSDP wrap policy.

Common situations: Changing the FSDP wrapping policy or auto_wrap hints so some parameters stop being FSDP-sharded while the loader's sharding bookkeeping still marks them rank-local; mixing FSDP and non-FSDP submodules; upgrading torch or model code where a param class no longer produces DTensors on meta init.

Related errors

• Rank-local TP shard produced for DTensor parameter {target_p
• GGUF diffusion checkpoints are incompatible with FSDP infere
• world_size must be positive and divide global_heads
• expected a tensor with at least one dimension
• dimension {dim} size {dim_size} must be divisible by 2 * gro

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5946ace4d986bad8. Report an issue: GitHub.