sgl-project/sglang · error · NotImplementedError

bitsandbytes 4-bit TP only supports column-parallel output s

Error message

bitsandbytes 4-bit TP only supports column-parallel output shards.

What it means

When sharding a bitsandbytes 4-bit quant state for tensor parallelism, only column-parallel output sharding is supported: the input dimension must be unsharded (input_start == 0 and local input dim equals full input dim). Row-parallel or input-sharded weights raise NotImplementedError.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/bitsandbytes.py:335

        state_by_shard = {0: quant_state}
        set_weight_attrs(param, {"bnb_quant_state": state_by_shard})
        offsets = torch.tensor([0, param.numel()]).cpu()
        set_weight_attrs(param, {"bnb_shard_offsets": offsets})


def _maybe_shard_bitsandbytes_4bit_quant_state(
    param: torch.nn.Parameter,
    quant_state: Any,
) -> Any:
    full_shape = tuple(getattr(param, "bnb_full_shape", tuple(quant_state.shape or ())))
    local_shape = tuple(getattr(param, "bnb_local_shape", full_shape))
    if not full_shape or local_shape == full_shape:
        return quant_state

    output_start = getattr(param, "bnb_output_shard_start", 0)
    input_start = getattr(param, "bnb_input_shard_start", 0)
    if input_start != 0 or local_shape[1] != full_shape[1]:
        raise NotImplementedError(
            "bitsandbytes 4-bit TP only supports column-parallel output shards."
        )
    if getattr(quant_state, "nested", False):
        raise NotImplementedError(
            "bitsandbytes 4-bit TP does not support nested quant states."
        )

    blocksize = quant_state.blocksize
    start_elem = output_start * full_shape[1]
    local_numel = local_shape[0] * local_shape[1]
    if start_elem % blocksize != 0 or local_numel % blocksize != 0:
        raise ValueError(
            "bitsandbytes 4-bit TP shard is not aligned to quantization blocks."
        )
    start_block = start_elem // blocksize
    num_blocks = local_numel // blocksize
    return type(quant_state)(
        absmax=quant_state.absmax.narrow(0, start_block, num_blocks).contiguous(),

View on GitHub (pinned to 0132848349)

Solutions

1. Mark the bnb-quantized linear layers as column-parallel so only the output dim shards
2. Replicate (TP=1) layers that must shard the input dimension
3. Use a quant method with full row-parallel support for those layers

if getattr(param, "bnb_input_shard_start", 0) != 0 or local_shape[1] != full_shape[1]:
    raise SystemExit("bnb 4-bit TP requires column-parallel sharding")

When it happens

Trigger: _maybe_shard_bitsandbytes_4bit_quant_state on a param whose bnb_input_shard_start != 0 or whose local_shape[1] != full_shape[1], i.e. the layer is row-parallel/sharded on the input dim.

Common situations: Serving a bnb 4-bit model with TP where a linear layer is configured RowParallelLinear instead of column-parallel; custom models mixing parallelism styles on quantized layers.

Related errors

• bitsandbytes 4-bit TP does not support nested quant states.
• bitsandbytes 4-bit TP shard is not aligned to quantization b
• GPTQ act_order on XPU requires each group_size block of inpu
• QVGPackedCausalKVCache does not support pinned-sink (longliv
• The input size is not aligned with the quantized weight shap

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/eb76d3a03ac6de4f. Report an issue: GitHub.