{"record":{"id":"063d2eac5107dda5","repo":"sgl-project/sglang","slug":"no-staged-embedding-for-send-req-id-req-id-alr","errorCode":null,"errorMessage":"no staged embedding for /send req_id={req_id} (already released)","messagePattern":"no staged embedding for /send req_id=(.+?) \\(already released\\)","errorType":"http","errorClass":"MMError","httpStatus":500,"severity":"error","filePath":"python/sglang/srt/disaggregation/encoder/runtime.py","lineNumber":1295,"sourceCode":"            try:\n                content = await server_module.meta_registry.wait(request[\"req_id\"])\n            except asyncio.TimeoutError as e:\n                raise MMError(\n                    \"encode metadata not ready\", code=HTTPStatus.GATEWAY_TIMEOUT\n                ) from e\n        elif dp_type == \"send\":\n            req_id = request[\"req_id\"]\n            sent = await enc.send(\n                req_id=req_id,\n                prefill_host=request[\"prefill_host\"],\n                embedding_port=request[\"embedding_port\"],\n                session_id=request[\"session_id\"],\n                buffer_address=request[\"buffer_address\"],\n            )\n            if not sent:\n                # Error envelope, not 200 + phantom count: the decoder must\n                # fail fast instead of waiting for a ZMQ ack that never comes.\n                raise MMError(\n                    f\"no staged embedding for /send req_id={req_id} \"\n                    f\"(already released)\"\n                )\n            # Releasing on the first /send breaks decoder TP > 1. No count means\n            # a pre-refcount decoder: stay eager rather than pin until the sweep.\n            receive_count = request.get(\"receive_count\")\n            if receive_count:\n                await server_module.meta_registry.note_send_done(req_id, receive_count)\n            else:\n                await enc.release_request(req_id)\n            content = None\n        else:\n            content = await execute_encode_pipeline(enc, sched, request)\n\n        logger.info(\n            f\"MM-Encoder [dp_rank={dp_rank}] {dp_type} done: \"\n            f\"req_id={request.get('req_id', '?')}, \"\n            f\"modality={request.get('modality', 'image')}, \"","sourceCodeStart":1277,"sourceCodeEnd":1313,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/encoder/runtime.py#L1277-L1313","documentation":"The decoder called the encoder's /send endpoint for a req_id whose staged embedding has already been released, so enc.send(...) returned False. The encoder deliberately returns an error envelope instead of 200 so the decoder fails fast rather than waiting forever for a ZMQ ack that will never arrive.","triggerScenarios":"Duplicate /send calls for the same req_id after the embedding was released (e.g. decoder TP ranks retrying, or a race where one rank's send triggers eager release and a later send arrives); sending after an explicit release_request; using a pre-refcount decoder that re-sends.","commonSituations":"Decoder TP > 1 with mismatched receive_count accounting; client retry logic re-issuing /send; a stale req_id reused across requests after the sweep released it.","solutions":["Make the decoder send exactly once per req_id (deduplicate retries across TP ranks)","Ensure all decoder TP ranks register receive_count correctly before the first /send so release is refcounted, not eager","Regenerate a fresh req_id for retried requests instead of reusing the old one","Check for duplicate delivery of the /send HTTP request (proxy retries, at-least-once middleware)"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"if encoder_client.already_sent(req_id):\n    skip_send(req_id)  # idempotency check before /send","typeGuard":null,"tryCatchPattern":"try:\n    await encoder.send(req_id)\nexcept MMError as e:\n    if 'already released' in str(e):\n        treat_as_duplicate(req_id); return  # not fatal for this rank\n    raise","preventionTips":["Send exactly once per req_id; dedupe across TP ranks","Always register receive_count before the first /send","Use fresh req_ids on client retries"],"tags":["encoder","send","request-lifecycle","race-condition"],"backgroundTag":"duplicate-request-after-release","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}