sgl-project/sglang · error · ConnectionError

Connection closed while reading message body

Error message

Connection closed while reading message body

What it means

Weight-cache IPC protocol error: the peer closed the connection after the message length header was read but before the full message body arrived. Typically indicates the daemon/client crashed or timed out mid-send.

Solutions

  1. Check daemon liveness and logs; restart it
  2. Ensure daemon has enough memory for the response payload
  3. Retry the fetch after a clean restart
Defensive patterns

Strategy: retry

Try / catch

try:
    msg = recv_msg(sock)
except ConnectionError as e:
    if 'message body' in str(e): reconnect_and_retry()

Prevention

When it happens

Trigger: Daemon killed or crashing while sending a fetch-state response; socket reset partway through a large pickled body.

Common situations: Daemon OOM-killed during large weight state transfer; network/unix socket closed mid-send.

Understand the failure class

Related errors


AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5a9a4c3734936abd. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/weight_cache/protocol.py:227

def send_msg(sock, obj: Any) -> None:
    """Send a length-prefixed pickled message over a socket."""
    data = pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL)
    header = struct.pack("!I", len(data))
    sock.sendall(header + data)


def recv_msg(sock) -> Any:
    """Receive a length-prefixed pickled message from a socket."""
    header = _recv_exact(sock, 4)
    if header is None:
        raise ConnectionError("Connection closed while reading message header")
    length = struct.unpack("!I", header)[0]
    if length > MAX_MSG_SIZE:
        raise ValueError(f"Message size {length} exceeds {MAX_MSG_SIZE} byte cap")
    data = _recv_exact(sock, length)
    if data is None:
        raise ConnectionError("Connection closed while reading message body")
    return safe_pickle_loads(data)


def _recv_exact(sock, n: int) -> Optional[bytes]:
    """Receive exactly n bytes from a socket."""
    buf = bytearray()
    while len(buf) < n:
        chunk = sock.recv(n - len(buf))
        if not chunk:
            return None
        buf.extend(chunk)
    return bytes(buf)


def compute_env_stamp() -> Dict[str, str]:
    """Local environment fingerprint for the IPC weight cache.

    Returns the device compute capability and torch version of the current

View on GitHub (pinned to 0132848349)