sgl-project/sglang · error · ConnectionError

Connection closed while reading message header

Error message

Connection closed while reading message header

What it means

recv_msg got zero bytes reading the 4-byte length prefix, meaning the peer closed the socket before sending a message. Part of the length-prefixed pickled protocol used between weight cache client/daemon.

Source

Thrown at python/sglang/srt/weight_cache/protocol.py:221

# Socket protocol helpers
# ---------------------------------------------------------------------------


MAX_MSG_SIZE = 256 * 1024 * 1024  # 256 MiB


def send_msg(sock, obj: Any) -> None:
    """Send a length-prefixed pickled message over a socket."""
    data = pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL)
    header = struct.pack("!I", len(data))
    sock.sendall(header + data)


def recv_msg(sock) -> Any:
    """Receive a length-prefixed pickled message from a socket."""
    header = _recv_exact(sock, 4)
    if header is None:
        raise ConnectionError("Connection closed while reading message header")
    length = struct.unpack("!I", header)[0]
    if length > MAX_MSG_SIZE:
        raise ValueError(f"Message size {length} exceeds {MAX_MSG_SIZE} byte cap")
    data = _recv_exact(sock, length)
    if data is None:
        raise ConnectionError("Connection closed while reading message body")
    return safe_pickle_loads(data)


def _recv_exact(sock, n: int) -> Optional[bytes]:
    """Receive exactly n bytes from a socket."""
    buf = bytearray()
    while len(buf) < n:
        chunk = sock.recv(n - len(buf))
        if not chunk:
            return None
        buf.extend(chunk)
    return bytes(buf)

View on GitHub (pinned to 0132848349)

Solutions

1. Check daemon logs for the crash cause and restart it
2. Verify the daemon is still running for this rank (ready file pid)
3. Retry the load or disable the weight cache

import os
def peer_alive(sock_path):
    return os.path.exists(sock_path)

try:
    msg = recv_msg(sock)
except ConnectionError:
    restart_daemon_and_retry()

When it happens

Trigger: Daemon process exits/crashes between connection accept and first write; peer calls close() after sending no data; called from _handle_connection, _fetch_from_cache, run_single_daemon, main.

Common situations: Weight cache daemon crashes during model load; daemon shuts down mid-handshake; test harness closes socket early.

Understand the failure class

• Connection failures: ECONNREFUSED, ECONNRESET, and friends — why connections get refused, reset, or dropped.

Related errors

• Connection closed while reading message body
• [IpcModelLoader] Error communicating with daemon at {self.so
• Unsupported socket type: {socket_type}
• Multi-node weight cache daemons (nnodes > 1) require --dist-
• Weight cache daemon for pp_rank={pp_rank} tp_rank={tp_rank}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/65a72fbec0dd1df3. Report an issue: GitHub.