sgl-project/sglang · error · ConnectionError

Connection closed while reading message body

Error message

Connection closed while reading message body

What it means

recv_msg read the length prefix but _recv_exact for the body returned None — the peer closed the connection mid-message. The framed message was truncated.

Source

Thrown at python/sglang/srt/weight_cache/protocol.py:227

def send_msg(sock, obj: Any) -> None:
    """Send a length-prefixed pickled message over a socket."""
    data = pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL)
    header = struct.pack("!I", len(data))
    sock.sendall(header + data)


def recv_msg(sock) -> Any:
    """Receive a length-prefixed pickled message from a socket."""
    header = _recv_exact(sock, 4)
    if header is None:
        raise ConnectionError("Connection closed while reading message header")
    length = struct.unpack("!I", header)[0]
    if length > MAX_MSG_SIZE:
        raise ValueError(f"Message size {length} exceeds {MAX_MSG_SIZE} byte cap")
    data = _recv_exact(sock, length)
    if data is None:
        raise ConnectionError("Connection closed while reading message body")
    return safe_pickle_loads(data)


def _recv_exact(sock, n: int) -> Optional[bytes]:
    """Receive exactly n bytes from a socket."""
    buf = bytearray()
    while len(buf) < n:
        chunk = sock.recv(n - len(buf))
        if not chunk:
            return None
        buf.extend(chunk)
    return bytes(buf)


def compute_env_stamp() -> Dict[str, str]:
    """Local environment fingerprint for the IPC weight cache.

    Returns the device compute capability and torch version of the current

View on GitHub (pinned to 0132848349)

Solutions

1. Check daemon liveness and logs; restart it
2. Ensure daemon has enough memory for the response payload
3. Retry the fetch after a clean restart

try:
    msg = recv_msg(sock)
except ConnectionError as e:
    if 'message body' in str(e): reconnect_and_retry()

When it happens

Trigger: Daemon killed or crashing while sending a fetch-state response; socket reset partway through a large pickled body.

Common situations: Daemon OOM-killed during large weight state transfer; network/unix socket closed mid-send.

Understand the failure class

• Connection failures: ECONNREFUSED, ECONNRESET, and friends — why connections get refused, reset, or dropped.

Related errors

• Connection closed while reading message header
• Unsupported socket type: {socket_type}
• CUDA VMM POSIX FD broker returned no file descriptor
• duplicate fd for {key}
• Could not bind port {port} on any configured address family

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5a9a4c3734936abd. Report an issue: GitHub.