sgl-project/sglang · critical · RuntimeError

Transfer thread failed because of {e}. Prefill instance with

Error message

Transfer thread failed because of {e}. Prefill instance with bootstrap_port={self.bootstrap_port} is dead.

What it means

transfer_worker wraps any exception escaping its loop and re-raises as RuntimeError declaring the prefill instance dead, including the bootstrap_port for identification. The original exception text is embedded in the message.

Source

Thrown at python/sglang/srt/disaggregation/mooncake/conn.py:1923

                        kv_chunk.is_last_chunk
                        and self.check_status(kv_chunk.room) == KVPoll.Failed
                    )
                ):
                    self._staging_outstanding.pop(kv_chunk.room, None)
                    if kv_chunk.room in self.transfer_infos:
                        self.transfer_infos.pop(kv_chunk.room)
                    self.req_to_decode_prefix_len.pop(kv_chunk.room, None)
                    if self.enable_staging:
                        # Purge prefetch bookkeeping for the finished room.
                        # Snapshot first: the scheduler thread adds concurrently.
                        for key in list(self._staging_ctx.prefetch_requested):
                            if key[0] == kv_chunk.room:
                                self._staging_ctx.prefetch_requested.discard(key)
                        self._staging_ctx.prefetched_rooms.discard(kv_chunk.room)

            except Exception as e:
                # NOTE(shangming): Remove this when we make sure the transfer thread is bug-free
                raise RuntimeError(
                    f"Transfer thread failed because of {e}. Prefill instance with bootstrap_port={self.bootstrap_port} is dead."
                )

    def start_prefill_thread(self):
        def bootstrap_thread():
            """This thread recvs pre-alloc notification from the decode engine"""
            # KVPoll.Bootstrapping -> KVPoll.WaitingForInput
            while True:
                waiting_req_bytes = self.server_socket.recv_multipart()
                room = waiting_req_bytes[0].decode("ascii")
                # Staging: decode reports consumption watermark back to prefill
                if room == "WATERMARK":
                    handle_watermark_msg(self._staging_ctx, waiting_req_bytes)
                    continue
                # Staging: decode replies with allocated staging offset
                if room == "STAGING_RSP":
                    handle_staging_rsp(waiting_req_bytes, self.transfer_infos)
                    continue

View on GitHub (pinned to 0132848349)

Solutions

1. Read the embedded {e} text and fix the underlying exception first (e.g. transfer_sync failure, index mismatch)
2. Check logs for the stack above this raise
3. Restart the prefill instance after fixing the root cause — it is declared dead intentionally

catch RuntimeError 'Transfer thread failed'; extract inner {e} cause, restart the prefill instance — it will not recover in-process

When it happens

Trigger: Any unhandled exception in the prefill transfer thread — RDMA errors, layout validation failures, staging bugs — gets wrapped here; the true cause is the inner {e}.

Common situations: A downstream error (transfer failure, index mismatch) surfacing at the thread boundary; the message itself is the wrapper, not the root cause.

Related errors

• {name}
• Page size mismatch: prefill server has page_size={info.page_
• KV cache dtype mismatch: prefill server has kv_cache_dtype={
• DSV4 HiSparse direct PD transfer currently requires the Moon
• mooncake encoder_transfer_backend requires HTTP encoders; us

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/62e6c9185317c76e. Report an issue: GitHub.