{"record":{"id":"eb76d3a03ac6de4f","repo":"sgl-project/sglang","slug":"bitsandbytes-4-bit-tp-only-supports-column-paralle","errorCode":null,"errorMessage":"bitsandbytes 4-bit TP only supports column-parallel output shards.","messagePattern":"bitsandbytes 4-bit TP only supports column-parallel output shards\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/quantization/bitsandbytes.py","lineNumber":335,"sourceCode":"        state_by_shard = {0: quant_state}\n        set_weight_attrs(param, {\"bnb_quant_state\": state_by_shard})\n        offsets = torch.tensor([0, param.numel()]).cpu()\n        set_weight_attrs(param, {\"bnb_shard_offsets\": offsets})\n\n\ndef _maybe_shard_bitsandbytes_4bit_quant_state(\n    param: torch.nn.Parameter,\n    quant_state: Any,\n) -> Any:\n    full_shape = tuple(getattr(param, \"bnb_full_shape\", tuple(quant_state.shape or ())))\n    local_shape = tuple(getattr(param, \"bnb_local_shape\", full_shape))\n    if not full_shape or local_shape == full_shape:\n        return quant_state\n\n    output_start = getattr(param, \"bnb_output_shard_start\", 0)\n    input_start = getattr(param, \"bnb_input_shard_start\", 0)\n    if input_start != 0 or local_shape[1] != full_shape[1]:\n        raise NotImplementedError(\n            \"bitsandbytes 4-bit TP only supports column-parallel output shards.\"\n        )\n    if getattr(quant_state, \"nested\", False):\n        raise NotImplementedError(\n            \"bitsandbytes 4-bit TP does not support nested quant states.\"\n        )\n\n    blocksize = quant_state.blocksize\n    start_elem = output_start * full_shape[1]\n    local_numel = local_shape[0] * local_shape[1]\n    if start_elem % blocksize != 0 or local_numel % blocksize != 0:\n        raise ValueError(\n            \"bitsandbytes 4-bit TP shard is not aligned to quantization blocks.\"\n        )\n    start_block = start_elem // blocksize\n    num_blocks = local_numel // blocksize\n    return type(quant_state)(\n        absmax=quant_state.absmax.narrow(0, start_block, num_blocks).contiguous(),","sourceCodeStart":317,"sourceCodeEnd":353,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/quantization/bitsandbytes.py#L317-L353","documentation":"When sharding a bitsandbytes 4-bit quant state for tensor parallelism, only column-parallel output sharding is supported: the input dimension must be unsharded (input_start == 0 and local input dim equals full input dim). Row-parallel or input-sharded weights raise NotImplementedError.","triggerScenarios":"_maybe_shard_bitsandbytes_4bit_quant_state on a param whose bnb_input_shard_start != 0 or whose local_shape[1] != full_shape[1], i.e. the layer is row-parallel/sharded on the input dim.","commonSituations":"Serving a bnb 4-bit model with TP where a linear layer is configured RowParallelLinear instead of column-parallel; custom models mixing parallelism styles on quantized layers.","solutions":["Mark the bnb-quantized linear layers as column-parallel so only the output dim shards","Replicate (TP=1) layers that must shard the input dimension","Use a quant method with full row-parallel support for those layers"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"if getattr(param, \"bnb_input_shard_start\", 0) != 0 or local_shape[1] != full_shape[1]:\n    raise SystemExit(\"bnb 4-bit TP requires column-parallel sharding\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use column-parallel layout for bnb 4-bit layers under TP","Avoid row-parallel quantized linears in this runtime"],"tags":["quantization","bitsandbytes","tensor-parallel","not-implemented"],"backgroundTag":"unsupported-tensor-parallel-sharding","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}