sgl-project/sglang · error · InternalError

Encoder produced {mm_embedding.shape[0]} tokens, but preproc

Error message

Encoder produced {mm_embedding.shape[0]} tokens, but preprocessor metadata expected {offset}

What it means

slice_embedding splits the encoder output into per-item slices using preprocessing-owned token_counts and validates that the total offset equals the encoder's produced token count. A mismatch means the encoder emitted a different number of multimodal tokens than the preprocessor predicted, indicating a serious encoder/preprocessor disagreement.

Source

Thrown at python/sglang/srt/disaggregation/encoder/server.py:784

                if val and int(val) > 0:
                    dims[Modality.AUDIO] = int(val)
                    break

        logger.info(f"Global cache embedding dims: {dims}")
        return dims

    def slice_embedding(
        self,
        mm_embedding: torch.Tensor,
        token_counts: Iterable[int],
    ) -> List[torch.Tensor]:
        """Slice embeddings using preprocessing-owned token counts."""
        slices, offset = [], 0
        for count in token_counts:
            slices.append(mm_embedding[offset : offset + count])
            offset += count
        if mm_embedding.shape[0] != offset:
            raise InternalError(
                f"Encoder produced {mm_embedding.shape[0]} tokens, but "
                f"preprocessor metadata expected {offset}"
            )
        return slices

    def _calculate_hashes_from_features(
        self, mm_feature, grid_thw: List, modality: Modality, mm_inputs=None
    ) -> List[int]:
        """CPU Task: Compute hashes based on processed feature patches."""
        preprocessed_items = (
            get_encoder_preprocessed_items(mm_inputs) if mm_inputs is not None else None
        )
        if preprocessed_items is not None:
            if len(preprocessed_items) != len(grid_thw):
                raise ValueError(
                    "Encoder preprocess item/grid mismatch: "
                    f"{len(preprocessed_items)} items != {len(grid_thw)} grids"
                )

View on GitHub (pinned to 0132848349)

Solutions

1. Verify encoder and preprocessor run the same model processor version and config (image resolutions, patch size, grid settings)
2. Reproduce with the failing image and compare token_counts from the preprocessor against the encoder's output shape
3. Fix the preprocessor's token counting (e.g. 2D grid logic) to match encoder behavior, or update the encoder config
4. Pin consistent package versions across encoder and decoder deployments

total = sum(token_counts)
assert mm_embedding.shape[0] == total, f'token drift: {mm_embedding.shape[0]} vs {total}'

When it happens

Trigger: Calling slice_embedding (via _encode_missing) where the vision encoder produced more or fewer tokens than the preprocessor's metadata predicted — e.g. different image resolution handling, patch/grid config mismatch (as in the Kimi-VL 2D grid case), or different processor versions on each side.

Common situations: Encoder and decoder/preprocessor using different versions of the multimodal processor or different image processing configs; a model integration where grid counting logic diverges from the encoder's actual behavior; resizing/cropping settings inconsistent between components.

Related errors

• Grid dim ({_mm_grid_attrs[modality]}) not found in {mm_input
• gRPC encode only supports IMAGE modality, got: {non_image}
• {error_msg}
• Feature attrs ({_mm_feature_attrs[modality]}) not found in {
• Encoder produced {mm_embedding.shape[0]} tokens, but preproc

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/802ff71023d5d4ac. Report an issue: GitHub.