{"record":{"id":"12c38bb1afed0ec1","repo":"sgl-project/sglang","slug":"encode-metadata-not-ready","errorCode":null,"errorMessage":"encode metadata not ready","messagePattern":"encode metadata not ready","errorType":"http","errorClass":"MMError","httpStatus":504,"severity":"error","filePath":"python/sglang/srt/disaggregation/encoder/runtime.py","lineNumber":1280,"sourceCode":") -> None:\n    t0 = time.time()\n    try:\n        if dp_type in (\"start_profile\", \"stop_profile\"):\n            content = await _dp_worker_handle_profile(enc, dp_rank, dp_type, request)\n        elif dp_type == \"health_encode\":\n            content = await _dp_worker_health_encode(enc)\n        elif dp_type == \"register_destinations\":\n            await enc.register_embedding_destinations(\n                request[\"req_id\"],\n                request[\"receive_count\"],\n                [request[\"receive_url\"]],\n            )\n            content = None\n        elif dp_type == \"wait_metadata\":\n            try:\n                content = await server_module.meta_registry.wait(request[\"req_id\"])\n            except asyncio.TimeoutError as e:\n                raise MMError(\n                    \"encode metadata not ready\", code=HTTPStatus.GATEWAY_TIMEOUT\n                ) from e\n        elif dp_type == \"send\":\n            req_id = request[\"req_id\"]\n            sent = await enc.send(\n                req_id=req_id,\n                prefill_host=request[\"prefill_host\"],\n                embedding_port=request[\"embedding_port\"],\n                session_id=request[\"session_id\"],\n                buffer_address=request[\"buffer_address\"],\n            )\n            if not sent:\n                # Error envelope, not 200 + phantom count: the decoder must\n                # fail fast instead of waiting for a ZMQ ack that never comes.\n                raise MMError(\n                    f\"no staged embedding for /send req_id={req_id} \"\n                    f\"(already released)\"\n                )","sourceCodeStart":1262,"sourceCodeEnd":1298,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/encoder/runtime.py#L1262-L1298","documentation":"The DP worker waited on meta_registry.wait(req_id) for the encoder-side metadata (token counts / hashes produced by preprocessing) to appear, but the wait timed out (asyncio.TimeoutError) and is converted to MMError with HTTP 504 GATEWAY_TIMEOUT. It means preprocessing metadata for this request never arrived at the encoder within the deadline.","triggerScenarios":"Sending a request with dp_type='wait_metadata' whose req_id was never registered by the preprocessor, or where registration is delayed beyond the registry's timeout — e.g. preprocess and encode phases racing, or the metadata message lost/dropped between components.","commonSituations":"Preprocessor service slow or crashed before publishing metadata; req_id mismatch between the prefill and encode paths; network/ZMQ delivery of metadata delayed under load; scaling up concurrency so metadata publication lags.","solutions":["Retry the request — transient scheduling lag often resolves it","Check that the preprocessor (scheduler/prefill side) successfully published metadata for this req_id","Increase the meta registry wait timeout if it is configurable","Reduce encoder concurrency or check for a stuck preprocessor worker"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"if not meta_registry.has(req_id):\n    await asyncio.sleep(backoff)  # allow publication to land before waiting","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    try:\n        return await worker.handle({'dp_type': 'wait_metadata', 'req_id': req_id})\n    except MMError as e:\n        if e.code != 504 or attempt == 2: raise\n        await asyncio.sleep(2 ** attempt)","preventionTips":["Publish metadata before dispatching the encode request","Alert on preprocessor lag/queue depth","Use idempotent req_ids so retries are safe"],"tags":["timeout","metadata","encoder","disaggregation"],"backgroundTag":"upstream-metadata-timeout","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}