{"record":{"id":"62e6c9185317c76e","repo":"sgl-project/sglang","slug":"transfer-thread-failed-because-of-e-prefill-ins","errorCode":null,"errorMessage":"Transfer thread failed because of {e}. Prefill instance with bootstrap_port={self.bootstrap_port} is dead.","messagePattern":"Transfer thread failed because of (.+?)\\. Prefill instance with bootstrap_port=(.+?) is dead\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/mooncake/conn.py","lineNumber":1923,"sourceCode":"                        kv_chunk.is_last_chunk\n                        and self.check_status(kv_chunk.room) == KVPoll.Failed\n                    )\n                ):\n                    self._staging_outstanding.pop(kv_chunk.room, None)\n                    if kv_chunk.room in self.transfer_infos:\n                        self.transfer_infos.pop(kv_chunk.room)\n                    self.req_to_decode_prefix_len.pop(kv_chunk.room, None)\n                    if self.enable_staging:\n                        # Purge prefetch bookkeeping for the finished room.\n                        # Snapshot first: the scheduler thread adds concurrently.\n                        for key in list(self._staging_ctx.prefetch_requested):\n                            if key[0] == kv_chunk.room:\n                                self._staging_ctx.prefetch_requested.discard(key)\n                        self._staging_ctx.prefetched_rooms.discard(kv_chunk.room)\n\n            except Exception as e:\n                # NOTE(shangming): Remove this when we make sure the transfer thread is bug-free\n                raise RuntimeError(\n                    f\"Transfer thread failed because of {e}. Prefill instance with bootstrap_port={self.bootstrap_port} is dead.\"\n                )\n\n    def start_prefill_thread(self):\n        def bootstrap_thread():\n            \"\"\"This thread recvs pre-alloc notification from the decode engine\"\"\"\n            # KVPoll.Bootstrapping -> KVPoll.WaitingForInput\n            while True:\n                waiting_req_bytes = self.server_socket.recv_multipart()\n                room = waiting_req_bytes[0].decode(\"ascii\")\n                # Staging: decode reports consumption watermark back to prefill\n                if room == \"WATERMARK\":\n                    handle_watermark_msg(self._staging_ctx, waiting_req_bytes)\n                    continue\n                # Staging: decode replies with allocated staging offset\n                if room == \"STAGING_RSP\":\n                    handle_staging_rsp(waiting_req_bytes, self.transfer_infos)\n                    continue","sourceCodeStart":1905,"sourceCodeEnd":1941,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/mooncake/conn.py#L1905-L1941","documentation":"transfer_worker wraps any exception escaping its loop and re-raises as RuntimeError declaring the prefill instance dead, including the bootstrap_port for identification. The original exception text is embedded in the message.","triggerScenarios":"Any unhandled exception in the prefill transfer thread — RDMA errors, layout validation failures, staging bugs — gets wrapped here; the true cause is the inner {e}.","commonSituations":"A downstream error (transfer failure, index mismatch) surfacing at the thread boundary; the message itself is the wrapper, not the root cause.","solutions":["Read the embedded {e} text and fix the underlying exception first (e.g. transfer_sync failure, index mismatch)","Check logs for the stack above this raise","Restart the prefill instance after fixing the root cause — it is declared dead intentionally"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch RuntimeError 'Transfer thread failed'; extract inner {e} cause, restart the prefill instance — it will not recover in-process","preventionTips":["Monitor transfer-thread liveness with a watchdog","Aggregate and alert on the inner causes, not this wrapper"],"tags":["pd-disagg","transfer-thread","mooncake","wrapper"],"backgroundTag":"background-thread-crash","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}