sgl-project/sglang · critical · InternalError

Mooncake transfer_sync failed for {req_id} (session={session

Error message

Mooncake transfer_sync failed for {req_id} (session={session_id}, nbytes={embedding.nbytes}, ret={xfer_ret})

What it means

The mooncake transfer engine's synchronous transfer (transfer_sync) returned a negative return code while pushing an embedding to the decode side. The message includes session id, byte count, and the engine return code for diagnosis.

Source

Thrown at python/sglang/srt/disaggregation/encoder/server.py:1569

            if not mr_already_registered:
                self.engine.register(embedding.data_ptr(), embedding.nbytes)
            _t_xfer_start = time.monotonic()
            xfer_ret = await asyncio.to_thread(
                self.engine.transfer_sync,
                session_id,
                embedding.data_ptr(),
                buffer_address,
                embedding.nbytes,
            )
            xfer_ms = (time.monotonic() - _t_xfer_start) * 1000.0
            if encoder_metrics_collector is not None:
                encoder_metrics_collector.observe_transfer(
                    xfer_ms / 1000.0, backend="mooncake"
                )
            if not mr_already_registered:
                self.engine.deregister(embedding.data_ptr())
            if xfer_ret < 0:
                raise InternalError(
                    f"Mooncake transfer_sync failed for {req_id} "
                    f"(session={session_id}, nbytes={embedding.nbytes}, "
                    f"ret={xfer_ret})"
                )
            # Emit at INFO for slow transfers or per-send registrations.
            if xfer_ms > 200.0 or not mr_already_registered:
                logger.info(
                    f"[{req_id}] mooncake transfer_sync={xfer_ms:.1f}ms "
                    f"nbytes={embedding.nbytes} shared_mr={mr_already_registered}"
                )

            # Sibling ranks re-read mm_data here; meta_registry owns the release.

        # Send ack/data
        if url is not None:
            endpoint = NetworkAddress.parse(url).to_tcp()
        else:
            endpoint = NetworkAddress(prefill_host, embedding_port).to_tcp()

View on GitHub (pinned to 0132848349)

Solutions

1. Retry the request (transient fabric errors often clear)
2. Check mooncake engine logs for the meaning of ret and the session state
3. Verify RDMA devices/interfaces are healthy on both ends (ibstat, connection to mooncake metadata service)
4. Restart the prefill/decode pair to re-establish sessions if the session was invalidated

catch InternalError containing 'transfer_sync failed'; inspect ret code, retry once, then tear down the session and fail the room

When it happens

Trigger: MooncakeStore.transfer_sync failing: broken RDMA connection, deregistered/expired session, fabric error, or wrong memory region — raised in _send after the bulk copy attempt.

Common situations: Decode/prefill instance restart mid-transfer, mooncake P2P fabric misconfiguration, network interruption, or buffer registration expiry under long uptime.

Related errors

• mooncake encoder_transfer_backend requires HTTP encoders; us
• [Staging] Bulk RDMA transfer failed with ret={ret}. src_ptr=
• NIXL transfer encountered ERR room={room}
• Mooncake Transfer Engine initialization failed.
• Failed to register buffer to Mooncake Store, error code: {re

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/65a1d7c9a26b3b5d. Report an issue: GitHub.