sgl-project/sglang · error · ValueError

shard_offset and shard_size must be provided

Error message

shard_offset and shard_size must be provided

What it means

load_merged_column_weight requires shard_offset and shard_size so it knows where within the merged column-parallel weight to place the loaded shard. Without them the write location is undefined, so the API raises ValueError immediately.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/parameter.py:118

    @property
    def output_dim(self):
        return self._output_dim

    def load_column_parallel_weight(self, loaded_weight: torch.Tensor) -> None:
        tp_rank = get_tp_rank()
        shard_size = self.data.shape[self.output_dim]
        loaded_weight = loaded_weight.narrow(
            self.output_dim, tp_rank * shard_size, shard_size
        )
        assert self.data.shape == loaded_weight.shape
        self.data.copy_(loaded_weight)

    def load_merged_column_weight(self, loaded_weight: torch.Tensor, **kwargs) -> None:

        shard_offset = kwargs.get("shard_offset")
        shard_size = kwargs.get("shard_size")
        if shard_offset is None or shard_size is None:
            raise ValueError("shard_offset and shard_size must be provided")
        if (
            isinstance(self, PackedColumnParameter | PackedvLLMParameter)
            and self.packed_dim == self.output_dim
        ):
            shard_size, shard_offset = self.adjust_shard_indexes_for_packing(
                shard_offset=shard_offset, shard_size=shard_size
            )

        param_data = self.data

        tp_rank = get_tp_rank()
        param_data = param_data.narrow(self.output_dim, shard_offset, shard_size)
        loaded_weight = loaded_weight.narrow(
            self.output_dim, tp_rank * shard_size, shard_size
        )
        assert param_data.shape == loaded_weight.shape
        param_data.copy_(loaded_weight)

View on GitHub (pinned to 0132848349)

Solutions

1. Pass shard_offset and shard_size describing where the shard sits in the merged dimension (e.g. shard_offset=0, shard_size=num_kv_heads*head_dim for the first shard)
2. If the tensor is not actually column-sharded, route it to load_weight / the plain Parameter loader instead of load_merged_column_weight
3. For PackedColumnParameter with packed_dim == output_dim, confirm the offsets are in unpacked units — the method will adjust them for packing

Example fix

// before
param.load_merged_column_weight(w)
// after
param.load_merged_column_weight(w, shard_offset=shard_id * shard_size, shard_size=shard_size)

if 'shard_offset' not in kwargs or 'shard_size' not in kwargs:
    param.load_weight(loaded_weight)  # non-merged path
else:
    param.load_merged_column_weight(loaded_weight, **kwargs)

def is_column_sharded(name: str, weights: dict) -> bool:
    return name in weights and isinstance(weights[name], tuple)  # (offset, size) entries

try:
    param.load_merged_column_weight(w, shard_offset=off, shard_size=sz)
except ValueError as e:
    logger.error("missing shard metadata for %s", param.name); raise

When it happens

Trigger: Calling parameter.load_merged_column_weight(loaded_weight) without passing shard_offset/shard_size kwargs — typically when a weight loader for a non-sharded (fully loaded) weight accidentally routes to the merged-column path, or a custom loader forgets the kwargs.

Common situations: Porting a vLLM-style model implementation whose weight_mapping omits shard metadata; checkpoints where a gate/up projection is stored as one fused tensor and the loader takes the 'no shard' branch; refactors of weight_loading_utils that drop kwargs.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Pi05 weight load failed: {len(missing)} missing weights, {mi
• qkv_proj weight {name}: unexpected shape {tuple(loaded_weigh
• scale_shift_table must have shape [9, D]
• expected a tensor with at least one dimension
• dimension {dim} size {dim_size} must be divisible by 2 * gro

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/eea4cc91866d8bd5. Report an issue: GitHub.