{"record":{"id":"d573c37875a3a3e1","repo":"sgl-project/sglang","slug":"no-embedding-available-for-mooncake-gpu-direct-tra","errorCode":null,"errorMessage":"No embedding available for Mooncake GPU-direct transfer: {req_id}","messagePattern":"No embedding available for Mooncake GPU-direct transfer: (.+?)","errorType":"http","errorClass":"InternalError","httpStatus":500,"severity":"critical","filePath":"python/sglang/srt/disaggregation/encoder/server.py","lineNumber":1538,"sourceCode":"                embedding_dim,\n            )\n            item_offset = item_end\n\n    async def _send(\n        self,\n        embedding: torch.Tensor,\n        mm_data: EmbeddingData,\n        session_id=None,\n        buffer_address=None,\n        prefill_host=None,\n        embedding_port=None,\n        url=None,\n    ):\n        if get_disagg().encoder_transfer_backend == \"mooncake\":\n            # Encode is synchronous, so mm_data was staged before /encode returned.\n            req_id = mm_data.req_id\n            if embedding is None:\n                raise InternalError(\n                    f\"No embedding available for Mooncake GPU-direct transfer: {req_id}\"\n                )\n\n            expected_nbytes = mm_data.shape[0] * mm_data.shape[1] * self._element_size\n            assert embedding.nbytes == expected_nbytes, (\n                f\"Embedding size mismatch for {req_id}: \"\n                f\"actual={embedding.nbytes}, expected={expected_nbytes} \"\n                f\"(shape={mm_data.shape}, element_size={self._element_size})\"\n            )\n\n            # Fall back to a per-send registration only if the shared one failed.\n            mr_already_registered = mm_data._mr_ptr == embedding.data_ptr()\n            if not mr_already_registered:\n                self.engine.register(embedding.data_ptr(), embedding.nbytes)\n            _t_xfer_start = time.monotonic()\n            xfer_ret = await asyncio.to_thread(\n                self.engine.transfer_sync,\n                session_id,","sourceCodeStart":1520,"sourceCodeEnd":1556,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/encoder/server.py#L1520-L1556","documentation":"On the mooncake GPU-direct transfer backend, _send requires a staged embedding tensor for the request; encoding is synchronous so the embedding must exist before /encode returned. If embedding is None the send cannot proceed and an InternalError is raised with the request id.","triggerScenarios":"Calling send() with embedding=None while encoder_transfer_backend == 'mooncake'; i.e. rank!=0 staged None, encoding failed silently, or send was called before _stage_embeddings ran.","commonSituations":"A bug in the encoder server staging path, calling send from a non-zero rank, or a race where send happens before the encode result is published.","solutions":["Ensure send() is only invoked on rank 0 after batch_encode returned a real embedding","Check encoder logs for a preceding encode failure that left embedding as None","Verify _stage_embeddings slice math produced tensors not None","If reproducible, file a bug — this is an internal invariant, not a config error"],"exampleFix":null,"handlingStrategy":"type-guard","validationCode":"if get_disagg().encoder_transfer_backend == \"mooncake\" and embedding is None:\n    raise RuntimeError(f\"encode produced no embedding for {mm_data.req_id}; refusing to send\")","typeGuard":"def has_embedding(e) -> bool: return e is not None and e.nbytes > 0","tryCatchPattern":"catch InternalError on send; mark request failed and re-encode once before dropping","preventionTips":["Only send from rank 0 after encode completes","Log embedding presence per req_id at debug"],"tags":["mooncake","disaggregation","encoder","gpu-direct"],"backgroundTag":"missing-required-value","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}