{"record":{"id":"65a72fbec0dd1df3","repo":"sgl-project/sglang","slug":"connection-closed-while-reading-message-header","errorCode":null,"errorMessage":"Connection closed while reading message header","messagePattern":"Connection closed while reading message header","errorType":"error_code","errorClass":"ConnectionError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/weight_cache/protocol.py","lineNumber":221,"sourceCode":"# Socket protocol helpers\n# ---------------------------------------------------------------------------\n\n\nMAX_MSG_SIZE = 256 * 1024 * 1024  # 256 MiB\n\n\ndef send_msg(sock, obj: Any) -> None:\n    \"\"\"Send a length-prefixed pickled message over a socket.\"\"\"\n    data = pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL)\n    header = struct.pack(\"!I\", len(data))\n    sock.sendall(header + data)\n\n\ndef recv_msg(sock) -> Any:\n    \"\"\"Receive a length-prefixed pickled message from a socket.\"\"\"\n    header = _recv_exact(sock, 4)\n    if header is None:\n        raise ConnectionError(\"Connection closed while reading message header\")\n    length = struct.unpack(\"!I\", header)[0]\n    if length > MAX_MSG_SIZE:\n        raise ValueError(f\"Message size {length} exceeds {MAX_MSG_SIZE} byte cap\")\n    data = _recv_exact(sock, length)\n    if data is None:\n        raise ConnectionError(\"Connection closed while reading message body\")\n    return safe_pickle_loads(data)\n\n\ndef _recv_exact(sock, n: int) -> Optional[bytes]:\n    \"\"\"Receive exactly n bytes from a socket.\"\"\"\n    buf = bytearray()\n    while len(buf) < n:\n        chunk = sock.recv(n - len(buf))\n        if not chunk:\n            return None\n        buf.extend(chunk)\n    return bytes(buf)","sourceCodeStart":203,"sourceCodeEnd":239,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/weight_cache/protocol.py#L203-L239","documentation":"recv_msg got zero bytes reading the 4-byte length prefix, meaning the peer closed the socket before sending a message. Part of the length-prefixed pickled protocol used between weight cache client/daemon.","triggerScenarios":"Daemon process exits/crashes between connection accept and first write; peer calls close() after sending no data; called from _handle_connection, _fetch_from_cache, run_single_daemon, main.","commonSituations":"Weight cache daemon crashes during model load; daemon shuts down mid-handshake; test harness closes socket early.","solutions":["Check daemon logs for the crash cause and restart it","Verify the daemon is still running for this rank (ready file pid)","Retry the load or disable the weight cache"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import os\ndef peer_alive(sock_path):\n    return os.path.exists(sock_path)","typeGuard":null,"tryCatchPattern":"try:\n    msg = recv_msg(sock)\nexcept ConnectionError:\n    restart_daemon_and_retry()","preventionTips":["Keep the daemon alive for the full load duration","Handle ConnectionError as a recoverable transport failure"],"tags":["protocol","socket","eof","weight-cache"],"backgroundTag":"connection-closed-by-peer","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}