{"record":{"id":"60482d5a321273df","repo":"sgl-project/sglang","slug":"kimi-k3-deferred-gpu-preprocessing-produced-wrong","errorCode":null,"errorMessage":"Kimi-K3 deferred GPU preprocessing produced wrong grids","messagePattern":"Kimi-K3 deferred GPU preprocessing produced wrong grids","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/kimi_k3.py","lineNumber":3461,"sourceCode":"                        image_scale, image_bias = normalization_tensors(\n                            first_config.image_mean,\n                            first_config.image_std,\n                            device,\n                        )\n                        pixel_values, produced_grids = _gpu_preprocess_images(\n                            [item.feature for item in group_items],\n                            [config.resize_config for config in group_configs],\n                            image_scale,\n                            image_bias,\n                            self.vision_tower.patch_size,\n                            to_chw=lambda image: to_chw_uint8(image, device=device),\n                            post_resize=lambda x: fill_transparent_bg(\n                                x, first_config.transparent_bg_config\n                            ),\n                        )\n                        expected_grids = grid_thws_host[global_indices]\n                        if not torch.equal(produced_grids.cpu(), expected_grids):\n                            raise ValueError(\n                                \"Kimi-K3 deferred GPU preprocessing produced wrong grids\"\n                            )\n                    elif backend == \"cpu\":\n                        from sglang.srt.multimodal.kimi_k3_image_processing import (\n                            materialize_kimi_k3_cpu_features,\n                        )\n\n                        pixel_values = materialize_kimi_k3_cpu_features(\n                            group_items, self._encoder_image_processor\n                        )\n                    else:\n                        raise ValueError(\n                            f\"Unsupported Kimi-K3 deferred preprocessing backend: {backend}\"\n                        )\n\n                    patch_counts = [\n                        int(grid_thws_host[index].prod().item())\n                        for index in global_indices","sourceCodeStart":3443,"sourceCodeEnd":3479,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/kimi_k3.py#L3443-L3479","documentation":"After deferred GPU preprocessing, the produced grid_thws must exactly equal the expected grids (grid_thws_host for those indices); a mismatch raises ValueError. This is a self-check that the GPU resize/patching pipeline reproduced the expected vision grids.","triggerScenarios":"Deferred GPU preprocessing (backend='gpu') runs for a group and torch.equal(produced_grids, expected_grids) is False — e.g. resize/post-resize (transparent background fill) behaving differently than the reference processor.","commonSituations":"Changes to image resize parameters, transparent_bg_config mismatches, GPU preprocessing kernel/version drift, or non-deterministic resizing on unusual image sizes.","solutions":["Verify resize config and transparent_bg_config match the HF reference processor","Update/re-pin the GPU preprocessing kernel so its grid computation matches the reference","As a workaround, route those items through the CPU backend (materialize_kimi_k3_cpu_features)","Report with the offending image and configs if it persists after config alignment"],"exampleFix":"// before\nbackend = \"gpu\"  # raises on grid mismatch\n\n// after\nbackend = \"cpu\"  # deterministic reference path while GPU grids are fixed","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    feats = model.get_image_feature(items)\nexcept ValueError as e:\n    if \"wrong grids\" in str(e):\n        mark_items_cpu_backend(items)\n        feats = model.get_image_feature(items)  # CPU reference path\n    else:\n        raise","preventionTips":["Pin the GPU preprocessing kernel version matched to the reference processor","Keep resize and transparent_bg_config identical to the HF processor config","Run golden-image tests comparing GPU vs CPU grids on each release"],"tags":["kimi-k3","gpu-preprocessing","grid-mismatch","sanity-check"],"backgroundTag":"preprocessing-output-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}