sgl-project/sglang · error · InternalError

Encoder produced {mm_embedding.shape[0]} tokens, but preproc

Error message

Encoder produced {mm_embedding.shape[0]} tokens, but preprocessor metadata expected {expected_tokens}

What it means

Raised by the disaggregated multimodal encoder server when the encoder backend returns an embedding tensor whose token count (rows) disagrees with the token counts recorded by the preprocessor for the same request. It is a sanity check in _compute_embedding that guards the contract between preprocessing metadata and the actual model output.

Source

Thrown at python/sglang/srt/disaggregation/encoder/server.py:1488

    async def _compute_embedding(
        self,
        ctx: EncodeContext,
        *,
        keep_on_gpu: bool,
    ) -> Optional[torch.Tensor]:
        """Compute one flattened request with global cache as an optional stage."""
        if ctx.use_global_cache:
            mm_embedding = await self._compute_global_cache_embedding(
                ctx, keep_on_gpu=keep_on_gpu
            )
        else:
            mm_embedding = await self._compute_direct_embedding(
                ctx, keep_on_gpu=keep_on_gpu
            )

        expected_tokens = sum(ctx.preprocess_result.token_counts)
        if mm_embedding is not None and mm_embedding.shape[0] != expected_tokens:
            raise InternalError(
                f"Encoder produced {mm_embedding.shape[0]} tokens, but "
                f"preprocessor metadata expected {expected_tokens}"
            )
        return mm_embedding

    async def _publish_preprocess_metadata(
        self, ctx: EncodeContext, requests: List[dict]
    ) -> None:
        """Publish each request's size after preprocessing, before model forward."""
        if self.rank != 0:
            return
        embedding_dim = self._embedding_dims[ctx.modality]
        item_offset = 0
        for request, item_count in zip(requests, ctx.items_per_req):
            item_end = item_offset + item_count
            token_count = sum(ctx.preprocess_result.token_counts[item_offset:item_end])
            req_id = request["req_id"]
            state = self._require_active_encode_state(req_id)

View on GitHub (pinned to 0132848349)

Solutions

1. Verify the preprocessor version matches the model checkpoint (same processor config)
2. Log ctx.preprocess_result.token_counts and mm_embedding.shape side by side for the failing request to find which item diverges
3. If a custom encoder is registered, audit its reshape/concatenation logic to emit exactly expected_tokens rows
4. Report upstream if stock models trigger it — indicates encoder/preprocessor contract bug

expected = sum(ctx.preprocess_result.token_counts)
assert mm_embedding is None or mm_embedding.shape[0] == expected, (mm_embedding.shape[0], expected)

catch sglang InternalError around batch_encode; log token_counts vs shape and fail the single request, not the server

When it happens

Trigger: Calling batch_encode (which awaits _compute_direct_embedding) where sum(ctx.preprocess_result.token_counts) != mm_embedding.shape[0]; e.g. a vision model whose patch/token math changed, or a custom encoder returning padded/truncated embeddings.

Common situations: Upgrading a multimodal model whose processor now emits different token counts, mismatched processor/model revisions, or a custom _compute_direct_embedding implementation that reshapes embeddings incorrectly.

Related errors

• Qwen-VL position_ids do not match the attention input
• gRPC encode only supports IMAGE modality, got: {non_image}
• {error_msg}
• encode metadata not ready
• Encoder produced {mm_embedding.shape[0]} tokens, but preproc

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8617796df97b1e11. Report an issue: GitHub.