{"record":{"id":"d9426b4341a30ef4","repo":"sgl-project/sglang","slug":"mm-inputs-where-only-some-items-are-precomputed","errorCode":null,"errorMessage":"MM inputs where only some items are precomputed.","messagePattern":"MM inputs where only some items are precomputed\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/managers/mm_schedule.py","lineNumber":115,"sourceCode":"            chunk = None\n        else:\n            req_embeddings = torch.concat(\n                [item.precomputed_embeddings for item in items_per_req]\n            )\n            chunk, _, _ = get_embedding_chunk(\n                embedding=req_embeddings,\n                extend_prefix_len=prefix_length[i],\n                extend_seq_len=extend_len,\n                items_offset=items_offset,\n            )\n\n        if chunk is None and len(items_per_req) > 1:\n            return None\n        precomputed_embeddings.append(chunk)\n\n    if any(feature is not None for feature in precomputed_embeddings):\n        if not all(feature is not None for feature in precomputed_embeddings):\n            raise NotImplementedError(\n                \"MM inputs where only some items are precomputed.\"\n            )\n\n        # Normalize device across chunks before concat.\n        target_device = next(\n            (t.device for t in precomputed_embeddings if t.is_cuda),\n            precomputed_embeddings[0].device,\n        )\n        precomputed_embeddings = [\n            t if t.device == target_device else t.to(target_device, non_blocking=True)\n            for t in precomputed_embeddings\n        ]\n        result = torch.concat(precomputed_embeddings)\n        # some models embedding is 3-dim, reshape it to 2-dim (similar to get_embedding_chunk)\n        result = result.reshape(-1, result.shape[-1])\n        return result\n    return None\n","sourceCodeStart":97,"sourceCodeEnd":133,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/managers/mm_schedule.py#L97-L133","documentation":"In the multimodal scheduling path, if some items in a multi-item MM request carry precomputed embeddings and others do not, SGLang cannot mix precomputed and to-be-computed features and raises NotImplementedError. All items must be uniformly precomputed or uniformly raw.","triggerScenarios":"A single request with multiple images where only some entries in the embedding payload are present (chunk None with len(items_per_req)>1 skips single-item fallback); partial caching of precomputed image embeddings.","commonSituations":"Building precomputed-embedding payloads where one image lacks its embedding due to an upstream miss; version change that altered embedding payload requirements for multi-image requests.","solutions":["Compute and include embeddings for ALL items in the request, or none of them","Audit the embedding generation step for silent per-item failures","For mixed workloads, split into separate requests: fully-precomputed and fully-raw"],"exampleFix":"// before\nreq = {'image_data': [img1, img2], 'precomputed': {'embeds': [e1]}}\n// after\nreq = {'image_data': [img1, img2], 'precomputed': {'embeds': [e1, e2]}}","handlingStrategy":"validation","validationCode":"embeds = precomputed.get('embeds')\nif embeds is not None:\n    assert len([e for e in embeds if e is not None]) in (0, n_mm_items)","typeGuard":null,"tryCatchPattern":"try:\n    get_embedding_and_mask(...)\nexcept NotImplementedError:\n    # fall back: send raw inputs without precomputed embeddings\n    ...","preventionTips":["All-or-nothing precomputed embedding payloads","Validate embedding generation had no silent per-item failures"],"tags":["sglang","multimodal","precomputed-embeddings","not-implemented"],"backgroundTag":"partial-precomputed-input","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}