sgl-project/sglang · error · NotImplementedError

T5 attention bias with bucketed positions is not yet tested

Error message

T5 attention bias with bucketed positions is not yet tested

What it means

This T5-style relative attention bias module only supports the non-bucketed setting (num_buckets < 0), in which case num_buckets defaults to max_distance. If a positive num_buckets is passed (bucketed relative positions), __init__ raises NotImplementedError because that path is untested in this port.

Source

Thrown at python/sglang/srt/models/phi4mm_utils.py:732

            the maximum distance for logarithmic bucketing after which all
            positions are in the same bucket.
        symmetric: bool
            Whether to use symmetric or asymmetric biases. symmetric=False uses
            2x number of bias params to distinguish L->R from R->L. This was
            found to be better for the encoder.
    """

    def __init__(self, num_heads, num_buckets=-1, max_distance=1000, symmetric=False):
        super().__init__()
        self.num_heads = num_heads
        self.num_buckets = num_buckets
        self.max_distance = max_distance
        self.symmetric = symmetric
        self._skip_bucketing = self.num_buckets < 0
        if self._skip_bucketing:
            self.num_buckets = max_distance
        else:
            raise NotImplementedError(
                "T5 attention bias with bucketed positions is not yet tested"
            )
        if not self.symmetric:
            self.num_buckets *= 2
        self.bias_values = nn.Embedding(self.num_buckets, self.num_heads)

    def forward(self, x):
        # instantiate bias compatible with shape of x
        maxpos = x.size(1)
        context_position = torch.arange(maxpos, device=x.device, dtype=torch.long)[
            :, None
        ]
        memory_position = torch.arange(maxpos, device=x.device, dtype=torch.long)[
            None, :
        ]
        relative_position = memory_position - context_position
        # clipping to a maximum distance using ops that play well with ONNX
        # export

View on GitHub (pinned to 0132848349)

Solutions

1. Set num_buckets to -1 (or omit it) so bucketing is skipped and num_buckets=max_distance is used
2. If bucketing is required, port the tested bucketing logic from NeMo's RelativePositionEmbedding before using it

Example fix

# before
RelativePositionEmbedding(num_buckets=32, max_distance=128, num_heads=h)  # NotImplementedError
# after
RelativePositionEmbedding(num_buckets=-1, max_distance=128, num_heads=h)

assert num_buckets < 0, "bucketed T5 relative position bias is not implemented; use num_buckets=-1"

def supports_bucketing(num_buckets: int) -> bool:
    return num_buckets < 0  # only non-bucketed path is implemented

When it happens

Trigger: Instantiating the relative position embedding with an explicit num_buckets >= 0, i.e. requesting classic T5 bucketed relative position bias.

Common situations: Copying T5/NeMo attention bias parameters (T5 uses num_buckets=32, max_distance=128) into a Phi-4-MM audio conformer config; porting models that rely on bucketing.

Related errors

• projection_cls = {projection_cls}, not implemented
• /v1/models ${response.status}
• Unsupported msgpack byte ${b}
• This browser does not support gzip stream decoding
• {self._op_label()}: no triton backend

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8666324969623bd0. Report an issue: GitHub.