sgl-project/sglang · error · ValueError

bitsandbytes 4-bit TP shard is not aligned to quantization b

Error message

bitsandbytes 4-bit TP shard is not aligned to quantization blocks.

What it means

For TP sharding of bnb 4-bit quant states, both the shard's start element offset and its element count must be multiples of the quant blocksize, so absmax scales can be narrowed block-wise. Otherwise the scale tensor slicing would be invalid.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/bitsandbytes.py:347

    if not full_shape or local_shape == full_shape:
        return quant_state

    output_start = getattr(param, "bnb_output_shard_start", 0)
    input_start = getattr(param, "bnb_input_shard_start", 0)
    if input_start != 0 or local_shape[1] != full_shape[1]:
        raise NotImplementedError(
            "bitsandbytes 4-bit TP only supports column-parallel output shards."
        )
    if getattr(quant_state, "nested", False):
        raise NotImplementedError(
            "bitsandbytes 4-bit TP does not support nested quant states."
        )

    blocksize = quant_state.blocksize
    start_elem = output_start * full_shape[1]
    local_numel = local_shape[0] * local_shape[1]
    if start_elem % blocksize != 0 or local_numel % blocksize != 0:
        raise ValueError(
            "bitsandbytes 4-bit TP shard is not aligned to quantization blocks."
        )
    start_block = start_elem // blocksize
    num_blocks = local_numel // blocksize
    return type(quant_state)(
        absmax=quant_state.absmax.narrow(0, start_block, num_blocks).contiguous(),
        shape=torch.Size(local_shape),
        code=quant_state.code,
        blocksize=quant_state.blocksize,
        quant_type=quant_state.quant_type,
        dtype=quant_state.dtype,
        offset=None,
        state2=None,
    )

View on GitHub (pinned to 0132848349)

Solutions

1. Pick a TP size such that the per-rank output shard is a multiple of the bnb blocksize (64)
2. Ensure full hidden dims are multiples of the blocksize before quantizing
3. Reduce TP until alignment holds

if (output_start * full_shape[1]) % quant_state.blocksize != 0 or (local_shape[0]*local_shape[1]) % quant_state.blocksize != 0:
    raise SystemExit("shard not blocksize-aligned; adjust TP degree")

When it happens

Trigger: A column-parallel shard where output_start * full_input_dim or local_numel is not divisible by quant_state.blocksize (typically 64).

Common situations: TP degree producing output shard sizes not aligned to the bnb blocksize; models with hidden dims not divisible by blocksize*tp.

Related errors

• The input size is not aligned with the quantized weight shap
• bitsandbytes 4-bit TP only supports column-parallel output s
• bitsandbytes 4-bit TP does not support nested quant states.
• Parameter {param_name} not found in the model.
• Weight input_size_per_partition = {input_size_per_partition}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3e4d5a17e5aa4210. Report an issue: GitHub.