sgl-project/sglang · error · NotImplementedError

bitsandbytes 4-bit TP does not support nested quant states.

Error message

bitsandbytes 4-bit TP does not support nested quant states.

What it means

bitsandbytes 4-bit TP sharding cannot handle nested quant states (quant_state.nested True, i.e. second-level/double quantization). The narrow-based shard reconstruction only works on flat quant states.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/bitsandbytes.py:339

def _maybe_shard_bitsandbytes_4bit_quant_state(
    param: torch.nn.Parameter,
    quant_state: Any,
) -> Any:
    full_shape = tuple(getattr(param, "bnb_full_shape", tuple(quant_state.shape or ())))
    local_shape = tuple(getattr(param, "bnb_local_shape", full_shape))
    if not full_shape or local_shape == full_shape:
        return quant_state

    output_start = getattr(param, "bnb_output_shard_start", 0)
    input_start = getattr(param, "bnb_input_shard_start", 0)
    if input_start != 0 or local_shape[1] != full_shape[1]:
        raise NotImplementedError(
            "bitsandbytes 4-bit TP only supports column-parallel output shards."
        )
    if getattr(quant_state, "nested", False):
        raise NotImplementedError(
            "bitsandbytes 4-bit TP does not support nested quant states."
        )

    blocksize = quant_state.blocksize
    start_elem = output_start * full_shape[1]
    local_numel = local_shape[0] * local_shape[1]
    if start_elem % blocksize != 0 or local_numel % blocksize != 0:
        raise ValueError(
            "bitsandbytes 4-bit TP shard is not aligned to quantization blocks."
        )
    start_block = start_elem // blocksize
    num_blocks = local_numel // blocksize
    return type(quant_state)(
        absmax=quant_state.absmax.narrow(0, start_block, num_blocks).contiguous(),
        shape=torch.Size(local_shape),
        code=quant_state.code,
        blocksize=quant_state.blocksize,
        quant_type=quant_state.quant_type,

View on GitHub (pinned to 0132848349)

Solutions

1. Re-quantize the checkpoint with double quantization disabled (bnb_4bit_use_double_quant=False)
2. Run with TP=1 so no quant-state sharding is attempted

if getattr(quant_state, "nested", False):
    raise SystemExit("disable bnb double quant or use TP=1")

When it happens

Trigger: Loading a bnb 4-bit checkpoint saved with nested (double) quantization enabled and then applying TP sharding via _maybe_shard_bitsandbytes_4bit_quant_state.

Common situations: bnb quantization_config with bnb_4bit_use_double_quant=True combined with tensor parallel serving.

Related errors

• bitsandbytes 4-bit TP only supports column-parallel output s
• bitsandbytes 4-bit TP shard is not aligned to quantization b
• GPTQ act_order on XPU requires each group_size block of inpu
• QVGPackedCausalKVCache does not support pinned-sink (longliv
• The input size is not aligned with the quantized weight shap

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/94a16016b6ae2986. Report an issue: GitHub.