sgl-project/sglang · error · NotImplementedError

Shard id with multiple indices is not supported in weight_lo

Error message

Shard id with multiple indices is not supported in weight_loader, please use weight_loader_v2 instead.

What it means

The v1 weight_loader cannot handle a tuple loaded_shard_id (a shard spanning multiple indices, e.g. fused or nested shards). If the param lacks a load_merged_column_weight hook to redirect to weight_loader_v2, it raises NotImplementedError telling you to use weight_loader_v2.

Source

Thrown at python/sglang/srt/layers/linear.py:585

            params_dtype=params_dtype,
            quant_config=quant_config,
            prefix=prefix,
            tp_rank=tp_rank,
            tp_size=tp_size,
            use_presharded_weights=use_presharded_weights,
        )
        self.prefix = prefix

    def weight_loader(
        self,
        param: Parameter,
        loaded_weight: torch.Tensor,
        loaded_shard_id: tuple[int, ...] | int | None = None,
    ):
        if isinstance(loaded_shard_id, tuple):
            if hasattr(param, "load_merged_column_weight"):
                return self.weight_loader_v2(param, loaded_weight, loaded_shard_id)
            raise NotImplementedError(
                "Shard id with multiple indices is not supported in weight_loader, "
                "please use weight_loader_v2 instead."
            )

        # Special case for GGUF
        # initialize GGUF param after we know the quantize type
        is_gguf_weight = getattr(param, "is_gguf_weight", False)
        is_gguf_weight_type = getattr(param, "is_gguf_weight_type", False)
        if is_gguf_weight_type:
            param.data[loaded_shard_id].copy_(loaded_weight)
            param.shard_weight_type[loaded_shard_id] = loaded_weight.item()
            return

        if is_gguf_weight:
            output_dim = getattr(param, "output_dim", None)
            shard_size = loaded_weight.size(output_dim) // self.tp_size
            start_idx = self.tp_rank * shard_size

View on GitHub (pinned to 0132848349)

Solutions

1. Call weight_loader_v2 instead, which natively supports tuple shard ids
2. Make the param a merged-column parameter (attach load_merged_column_weight) so v1 auto-delegates
3. Update the model/quant code to stop constructing tuple shard ids for the v1 loader

Example fix

# before
linear.weight_loader(param, loaded_weight, loaded_shard_id=(1, 2))  # NotImplementedError

# after
linear.weight_loader_v2(param, loaded_weight, loaded_shard_id=(1, 2))

if isinstance(loaded_shard_id, tuple):
    assert hasattr(param, "load_merged_column_weight"), "v1 loader cannot take tuple shard ids"
    linear.weight_loader_v2(param, loaded_weight, loaded_shard_id)
else:
    linear.weight_loader(param, loaded_weight, loaded_shard_id)

def needs_v2_loader(param, loaded_shard_id) -> bool:
    return isinstance(loaded_shard_id, tuple) and not hasattr(param, "load_merged_column_weight")

try:
    linear.weight_loader(param, loaded_weight, loaded_shard_id)
except NotImplementedError as e:
    if "weight_loader_v2" in str(e):
        linear.weight_loader_v2(param, loaded_weight, loaded_shard_id)
    else:
        raise

When it happens

Trigger: Calling weight_loader(param, loaded_weight, loaded_shard_id=(i, j)) on a linear whose parameter is not a merged-column param (no load_merged_column_weight attribute) — e.g. custom quant methods or model code invoking the v1 loader with multi-index shard ids.

Common situations: Custom quantization or model classes overriding/forwarding to v1 weight_loader while the checkpoint uses fused shards; new model implementations passing tuple shard ids to the old loader API.

Related errors

• Unknown Shard Id {shard_id}
• Expected scalar scale for fused-in-checkpoint merged-column
• expected a tensor with at least one dimension
• dimension {dim} size {dim_size} must be divisible by 2 * gro
• f"Invalid prefix for SparseVideoGen2AttentionImpl: {prefix}"

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c02a2cbcc41152cf. Report an issue: GitHub.