{"record":{"id":"94a16016b6ae2986","repo":"sgl-project/sglang","slug":"bitsandbytes-4-bit-tp-does-not-support-nested-quan","errorCode":null,"errorMessage":"bitsandbytes 4-bit TP does not support nested quant states.","messagePattern":"bitsandbytes 4-bit TP does not support nested quant states\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/quantization/bitsandbytes.py","lineNumber":339,"sourceCode":"\n\ndef _maybe_shard_bitsandbytes_4bit_quant_state(\n    param: torch.nn.Parameter,\n    quant_state: Any,\n) -> Any:\n    full_shape = tuple(getattr(param, \"bnb_full_shape\", tuple(quant_state.shape or ())))\n    local_shape = tuple(getattr(param, \"bnb_local_shape\", full_shape))\n    if not full_shape or local_shape == full_shape:\n        return quant_state\n\n    output_start = getattr(param, \"bnb_output_shard_start\", 0)\n    input_start = getattr(param, \"bnb_input_shard_start\", 0)\n    if input_start != 0 or local_shape[1] != full_shape[1]:\n        raise NotImplementedError(\n            \"bitsandbytes 4-bit TP only supports column-parallel output shards.\"\n        )\n    if getattr(quant_state, \"nested\", False):\n        raise NotImplementedError(\n            \"bitsandbytes 4-bit TP does not support nested quant states.\"\n        )\n\n    blocksize = quant_state.blocksize\n    start_elem = output_start * full_shape[1]\n    local_numel = local_shape[0] * local_shape[1]\n    if start_elem % blocksize != 0 or local_numel % blocksize != 0:\n        raise ValueError(\n            \"bitsandbytes 4-bit TP shard is not aligned to quantization blocks.\"\n        )\n    start_block = start_elem // blocksize\n    num_blocks = local_numel // blocksize\n    return type(quant_state)(\n        absmax=quant_state.absmax.narrow(0, start_block, num_blocks).contiguous(),\n        shape=torch.Size(local_shape),\n        code=quant_state.code,\n        blocksize=quant_state.blocksize,\n        quant_type=quant_state.quant_type,","sourceCodeStart":321,"sourceCodeEnd":357,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/quantization/bitsandbytes.py#L321-L357","documentation":"bitsandbytes 4-bit TP sharding cannot handle nested quant states (quant_state.nested True, i.e. second-level/double quantization). The narrow-based shard reconstruction only works on flat quant states.","triggerScenarios":"Loading a bnb 4-bit checkpoint saved with nested (double) quantization enabled and then applying TP sharding via _maybe_shard_bitsandbytes_4bit_quant_state.","commonSituations":"bnb quantization_config with bnb_4bit_use_double_quant=True combined with tensor parallel serving.","solutions":["Re-quantize the checkpoint with double quantization disabled (bnb_4bit_use_double_quant=False)","Run with TP=1 so no quant-state sharding is attempted"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"if getattr(quant_state, \"nested\", False):\n    raise SystemExit(\"disable bnb double quant or use TP=1\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Quantize with bnb_4bit_use_double_quant=False when serving with TP"],"tags":["quantization","bitsandbytes","nested-quant","tensor-parallel","not-implemented"],"backgroundTag":"unsupported-tensor-parallel-sharding","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}