sgl-project/sglang · error · NotImplementedError

projection_cls = {projection_cls}, not implemented

Error message

projection_cls = {projection_cls}, not implemented

What it means

The audio-to-LLM projection module in Phi-4-MM audio only implements specific projection classes (a stack of Linear+GELU layers, and the audio_projection_for_vision variant). If projection_cls in the config does not match one of the implemented branches, __init__ raises NotImplementedError naming the offending class.

Source

Thrown at python/sglang/srt/models/phi4mm_audio.py:1173

            depth = 2
            self.linear_downsample_rate = (
                1 if (self.qformer or self.conv_ds) else self.downsample_rate
            )
            layers = [
                nn.Linear(audio_dim_out * self.linear_downsample_rate, dim_projection)
            ]
            for _ in range(1, depth):
                layers.extend([nn.GELU(), nn.Linear(dim_projection, dim_projection)])
            self.audio_projection = nn.Sequential(*layers)
            # NOTE vision-speech tasks use a separate projection layer
            layers = [
                nn.Linear(audio_dim_out * self.linear_downsample_rate, dim_projection)
            ]
            for _ in range(1, depth):
                layers.extend([nn.GELU(), nn.Linear(dim_projection, dim_projection)])
            self.audio_projection_for_vision = nn.Sequential(*layers)
        else:
            raise NotImplementedError(
                f"projection_cls = {projection_cls}, not implemented"
            )

        # TODO: audio sequence compression - Qformer
        self.vocab_size = config.vocab_size
        self.input_embeds = None
        self.audio_embed_sizes = None

    def set_audio_embeds(self, input_embeds: torch.FloatTensor) -> None:
        self.input_embeds = input_embeds

    def set_audio_embed_sizes(self, audio_embed_sizes: torch.LongTensor) -> None:
        self.audio_embed_sizes = audio_embed_sizes

    def get_audio_features(
        self,
        input_embeds: torch.FloatTensor,
        audio_attention_mask: torch.Tensor = None,

View on GitHub (pinned to 0132848349)

Solutions

1. Set projection_cls back to the supported value in the model/config
2. Use the original phi-4-multimodal-instruct checkpoint whose projection matches the implementation
3. Implement the missing projection class in phi4mm_audio.py following the nn.Sequential pattern of the existing branches

Example fix

// before
"projection_cls": "qformer"  # raises NotImplementedError
// after
"projection_cls": "linear"   # supported stack of Linear/GELU layers

SUPPORTED_PROJECTIONS = {"linear", "mlp"}  # values handled by phi4mm_audio branches
assert config.projection_cls in SUPPORTED_PROJECTIONS, f"projection_cls {config.projection_cls} unsupported"

def is_supported_projection(cls_name: str) -> bool:
    return cls_name in {"linear", "mlp"}

When it happens

Trigger: Instantiating the Phi4MMAudioModel / projection with a config whose projection_cls is anything other than the supported values (e.g. 'qformer' or a custom class name), since the Qformer path is an explicit TODO.

Common situations: Loading a fine-tuned or newer Phi-4-MM checkpoint that swapped the linear projection for a Q-Former-style compressor; hand-editing model configs.

Related errors

• Attention backend '{selected_backend}' is not supported by t
• Norm type {self.norm_type} not implemented
• Unknown image_vae_encoding_position: {image_vae_encoding_pos
• memory_position_mode must be one of {'reference', 'legacy',
• gRPC encode only supports IMAGE modality, got: {non_image}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/383e15b2a5305856. Report an issue: GitHub.