{"record":{"id":"a5532e118ab735a6","repo":"jingyaogong/minimind","slug":"sglang-update-policy-failed","errorCode":null,"errorMessage":"SGLang update_policy failed","messagePattern":"SGLang update_policy failed","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"trainer/rollout_engine.py","lineNumber":193,"sourceCode":" def update_policy(self, model: torch.nn.Module):\n ok = True\n if not dist.is_initialized() or dist.get_rank() == 0:\n try:\n unwrapped = model.module if isinstance(model, DistributedDataParallel) else model\n unwrapped = getattr(unwrapped, '_orig_mod', unwrapped)\n abs_path = os.path.abspath(self.shared_ckpt_path)\n state_dict = {k: v.detach().half().cpu() for k, v in unwrapped.state_dict().items()}\n unwrapped.save_pretrained(abs_path, state_dict=state_dict, safe_serialization=False)\n self.tokenizer.save_pretrained(abs_path)\n resp = self.http.post(f\"{self.base_url}/update_weights_from_disk\", json={\"model_path\": abs_path}, timeout=self.timeout)\n if resp.status_code != 200: print(f\"[SGLANG WARNING] update_weights 失败: {resp.status_code}, {resp.text}\")\n ok = resp.status_code == 200\n except Exception as e:\n print(f\"[SGLANG WARNING] update_weights 异常: {e}\"); ok = False\n if dist.is_initialized():\n ok_t = torch.tensor(int(ok), device=next(model.parameters()).device)\n dist.broadcast(ok_t, src=0); dist.barrier(); ok = bool(ok_t.item())\n if not ok: raise RuntimeError(\"SGLang update_policy failed\")\n return ok\n \n def flush_cache(self) -> bool:\n resp = self.http.post(f\"{self.base_url}/flush_cache\", timeout=30)\n return resp.status_code == 200\n \n def health(self) -> bool:\n try:\n resp = self.http.get(f\"{self.base_url}/health\", timeout=5)\n return resp.status_code == 200\n except:\n return False\n\n\n# ===== 工厂函数 =====\ndef create_rollout_engine(\n engine_type: str = \"torch\",\n policy_model: torch.nn.Module = None,","sourceCodeStart":175,"sourceCodeEnd":211,"githubUrl":"https://github.com/jingyaogong/minimind/blob/393e387e9ad99f0f04c296e4c5e7353f4444629f/trainer/rollout_engine.py#L175-L211","documentation":"RuntimeError raised by SGLangRolloutEngine.update_policy after the training process saves updated weights (save_pretrained to the shared checkpoint dir, half precision, safe_serialization=False) and then POSTs to the SGLang server's /update_weights_from_disk endpoint. It fires when the POST returns non-200 (printed as '[SGLANG WARNING] update_weights failed:

, ') or throws (printed as 'update_weights exception'), and the failure flag has been broadcast from rank 0 to all distributed ranks via dist.broadcast, so every rank raises together. It is a hard stop: the RL loop (PPO/GRPO-style weight sync) cannot continue with stale policy weights.","triggerScenarios":"Calling update_policy() while training multi-process (dist.is_initialized) after a gradient step: rank 0 writes the checkpoint to sglang_shared_path and calls POST {base_url}/update_weights_from_disk with json {'model_path': abs_path}. It fails when the SGLang server is down/unreachable, when the path is not visible to the server process (different mount/container), when save_pretrained partially wrote (no atomic rename), when the server rejects the non-safetensors pickle format, or on update timeout (self.timeout). Non-200 prints a WARNING with resp.text; exceptions print the exception; both set ok=False which is broadcast, then raise.","commonSituations":"SGLang server and trainer in separate containers/NFS mounts where the 'shared' checkpoint dir is not actually shared; SGLang version that lacks or renamed /update_weights_from_disk; disk full or permission error while writing the checkpoint; race where the server reads weights mid-write because safe_serialization=False writes a pickle non-atomically; timeout too small for large models; rank>0 processes raising because rank 0 failed and the broadcast propagated ok=0.","solutions":["Check the printed WARNING line immediately above the raise — it contains resp.status_code and resp.text from SGLang, which names the real cause.","Confirm the SGLang server is alive: engine.health() (GET {base_url}/health) and that base_url is reachable from the trainer process.","Make sglang_shared_path truly shared and writable by both processes (same host dir, or correctly mounted volume in both containers) and verify with os.path.exists after save.","Retry the sync once after flush_cache(): stale KV cache or a busy server can reject the update; POST /flush_cache then re-run update_policy.","Bump self.timeout — loading large half-precision weights from disk can exceed a short HTTP timeout.","Upgrade/align SGLang version so /update_weights_from_disk is supported, and prefer safe_serialization=True if the server requires safetensors.","Write atomically (save to temp dir, os.replace/rename to final path) so the server never loads a half-written checkpoint."],"exampleFix":"# before\nresp = self.http.post(f\"{self.base_url}/update_weights_from_disk\", json={\"model_path\": abs_path}, timeout=self.timeout)\nif resp.status_code != 200: print(f\"[SGLANG WARNING] update_weights failed: {resp.status_code}, {resp.text}\")\nok = resp.status_code == 200\n\n# after\ntmp_path = abs_path + '.tmp'\nunwrapped.save_pretrained(tmp_path, state_dict=state_dict, safe_serialization=False)\nself.tokenizer.save_pretrained(tmp_path)\nif os.path.exists(abs_path): shutil.rmtree(abs_path)\nos.rename(tmp_path, abs_path)\nresp = self.http.post(f\"{self.base_url}/update_weights_from_disk\", json={\"model_path\": abs_path}, timeout=self.timeout)\nok = resp.status_code == 200\nif not ok:\n    self.flush_cache()\n    resp = self.http.post(f\"{self.base_url}/update_weights_from_disk\", json={\"model_path\": abs_path}, timeout=self.timeout)\n    ok = resp.status_code == 200","handlingStrategy":"try-catch","validationCode":"# Pre-flight before update_policy(): server up + shared path visible + writable\nassert engine.health(), 'SGLang server not healthy'\nshared = os.path.abspath(engine.shared_ckpt_path)\nassert os.path.isdir(shared), f'shared path missing on trainer: {shared}'\nprobe = os.path.join(shared, '.probe')\nwith open(probe, 'w') as f: f.write('x')\nos.remove(probe)  # trainer side writable; confirm same volume is mounted in the server container too","typeGuard":null,"tryCatchPattern":"try:\n    ok = engine.update_policy()\nexcept RuntimeError as e:\n    if 'SGLang update_policy failed' in str(e):\n        # the real cause was printed as '[SGLANG WARNING] update_weights ...' on rank 0;\n        # flush stale cache and retry once before aborting the RL step\n        engine.flush_cache()\n        ok = engine.update_policy()\n        if not ok:\n            raise\n    else:\n        raise","preventionTips":["Health-check the SGLang server (GET /health) at the start of every training run, not just at update time.","Use a genuinely shared volume (same host path or correctly mounted NFS) for the checkpoint dir; test write+read from both trainer and server containers.","Save checkpoints atomically (temp dir + rename) so the server never loads partially written weights.","Set the update timeout proportional to model size; a timeout manifests as this same RuntimeError.","Keep trainer and SGLang versions in lockstep so /update_weights_from_disk semantics do not drift.","Treat the printed WARNING text as part of your monitoring — it carries resp.status_code and resp.text from SGLang."],"tags":["sglang","rl-training","weight-sync","distributed","checkpoint"],"backgroundTag":null,"analyzedSha":"393e387e9ad99f0f04c296e4c5e7353f4444629f","analyzedAt":"2026-08-15T03:55:47.817Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}