{"record":{"id":"3e4d5a17e5aa4210","repo":"sgl-project/sglang","slug":"bitsandbytes-4-bit-tp-shard-is-not-aligned-to-quan","errorCode":null,"errorMessage":"bitsandbytes 4-bit TP shard is not aligned to quantization blocks.","messagePattern":"bitsandbytes 4-bit TP shard is not aligned to quantization blocks\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/quantization/bitsandbytes.py","lineNumber":347,"sourceCode":"    if not full_shape or local_shape == full_shape:\n        return quant_state\n\n    output_start = getattr(param, \"bnb_output_shard_start\", 0)\n    input_start = getattr(param, \"bnb_input_shard_start\", 0)\n    if input_start != 0 or local_shape[1] != full_shape[1]:\n        raise NotImplementedError(\n            \"bitsandbytes 4-bit TP only supports column-parallel output shards.\"\n        )\n    if getattr(quant_state, \"nested\", False):\n        raise NotImplementedError(\n            \"bitsandbytes 4-bit TP does not support nested quant states.\"\n        )\n\n    blocksize = quant_state.blocksize\n    start_elem = output_start * full_shape[1]\n    local_numel = local_shape[0] * local_shape[1]\n    if start_elem % blocksize != 0 or local_numel % blocksize != 0:\n        raise ValueError(\n            \"bitsandbytes 4-bit TP shard is not aligned to quantization blocks.\"\n        )\n    start_block = start_elem // blocksize\n    num_blocks = local_numel // blocksize\n    return type(quant_state)(\n        absmax=quant_state.absmax.narrow(0, start_block, num_blocks).contiguous(),\n        shape=torch.Size(local_shape),\n        code=quant_state.code,\n        blocksize=quant_state.blocksize,\n        quant_type=quant_state.quant_type,\n        dtype=quant_state.dtype,\n        offset=None,\n        state2=None,\n    )\n","sourceCodeStart":329,"sourceCodeEnd":362,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/quantization/bitsandbytes.py#L329-L362","documentation":"For TP sharding of bnb 4-bit quant states, both the shard's start element offset and its element count must be multiples of the quant blocksize, so absmax scales can be narrowed block-wise. Otherwise the scale tensor slicing would be invalid.","triggerScenarios":"A column-parallel shard where output_start * full_input_dim or local_numel is not divisible by quant_state.blocksize (typically 64).","commonSituations":"TP degree producing output shard sizes not aligned to the bnb blocksize; models with hidden dims not divisible by blocksize*tp.","solutions":["Pick a TP size such that the per-rank output shard is a multiple of the bnb blocksize (64)","Ensure full hidden dims are multiples of the blocksize before quantizing","Reduce TP until alignment holds"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"if (output_start * full_shape[1]) % quant_state.blocksize != 0 or (local_shape[0]*local_shape[1]) % quant_state.blocksize != 0:\n    raise SystemExit(\"shard not blocksize-aligned; adjust TP degree\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pick TP degrees that keep per-rank shards multiples of blocksize (64)"],"tags":["quantization","bitsandbytes","tensor-parallel","alignment"],"backgroundTag":"unsupported-tensor-parallel-sharding","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}