What causes "SGLang update_policy failed" in jingyaogong/minimind?

Check the printed WARNING line immediately above the raise — it contains resp.status_code and resp.text from SGLang, which names the real cause.

jingyaogong/minimind · critical · RuntimeError

SGLang update_policy failed

Error message

SGLang update_policy failed

What it means

RuntimeError raised by SGLangRolloutEngine.update_policy after the training process saves updated weights (save_pretrained to the shared checkpoint dir, half precision, safe_serialization=False) and then POSTs to the SGLang server's /update_weights_from_disk endpoint. It fires when the POST returns non-200 (printed as '[SGLANG WARNING] update_weights failed: <code>, <body>') or throws (printed as 'update_weights exception'), and the failure flag has been broadcast from rank 0 to all distributed ranks via dist.broadcast, so every rank raises together. It is a hard stop: the RL loop (PPO/GRPO-style weight sync) cannot continue with stale policy weights.

Source

Thrown at trainer/rollout_engine.py:193

    def update_policy(self, model: torch.nn.Module):
        ok = True
        if not dist.is_initialized() or dist.get_rank() == 0:
            try:
                unwrapped = model.module if isinstance(model, DistributedDataParallel) else model
                unwrapped = getattr(unwrapped, '_orig_mod', unwrapped)
                abs_path = os.path.abspath(self.shared_ckpt_path)
                state_dict = {k: v.detach().half().cpu() for k, v in unwrapped.state_dict().items()}
                unwrapped.save_pretrained(abs_path, state_dict=state_dict, safe_serialization=False)
                self.tokenizer.save_pretrained(abs_path)
                resp = self.http.post(f"{self.base_url}/update_weights_from_disk", json={"model_path": abs_path}, timeout=self.timeout)
                if resp.status_code != 200: print(f"[SGLANG WARNING] update_weights 失败: {resp.status_code}, {resp.text}")
                ok = resp.status_code == 200
            except Exception as e:
                print(f"[SGLANG WARNING] update_weights 异常: {e}"); ok = False
        if dist.is_initialized():
            ok_t = torch.tensor(int(ok), device=next(model.parameters()).device)
            dist.broadcast(ok_t, src=0); dist.barrier(); ok = bool(ok_t.item())
        if not ok: raise RuntimeError("SGLang update_policy failed")
        return ok
    
    def flush_cache(self) -> bool:
        resp = self.http.post(f"{self.base_url}/flush_cache", timeout=30)
        return resp.status_code == 200
    
    def health(self) -> bool:
        try:
            resp = self.http.get(f"{self.base_url}/health", timeout=5)
            return resp.status_code == 200
        except:
            return False


# ===== 工厂函数 =====
def create_rollout_engine(
    engine_type: str = "torch",
    policy_model: torch.nn.Module = None,

View on GitHub (pinned to 393e387e9a)

Solutions

Check the printed WARNING line immediately above the raise — it contains resp.status_code and resp.text from SGLang, which names the real cause.
Confirm the SGLang server is alive: engine.health() (GET {base_url}/health) and that base_url is reachable from the trainer process.
Make sglang_shared_path truly shared and writable by both processes (same host dir, or correctly mounted volume in both containers) and verify with os.path.exists after save.
Retry the sync once after flush_cache(): stale KV cache or a busy server can reject the update; POST /flush_cache then re-run update_policy.
Bump self.timeout — loading large half-precision weights from disk can exceed a short HTTP timeout.
Upgrade/align SGLang version so /update_weights_from_disk is supported, and prefer safe_serialization=True if the server requires safetensors.
Write atomically (save to temp dir, os.replace/rename to final path) so the server never loads a half-written checkpoint.

Example fix

# before
resp = self.http.post(f"{self.base_url}/update_weights_from_disk", json={"model_path": abs_path}, timeout=self.timeout)
if resp.status_code != 200: print(f"[SGLANG WARNING] update_weights failed: {resp.status_code}, {resp.text}")
ok = resp.status_code == 200

# after
tmp_path = abs_path + '.tmp'
unwrapped.save_pretrained(tmp_path, state_dict=state_dict, safe_serialization=False)
self.tokenizer.save_pretrained(tmp_path)
if os.path.exists(abs_path): shutil.rmtree(abs_path)
os.rename(tmp_path, abs_path)
resp = self.http.post(f"{self.base_url}/update_weights_from_disk", json={"model_path": abs_path}, timeout=self.timeout)
ok = resp.status_code == 200
if not ok:
    self.flush_cache()
    resp = self.http.post(f"{self.base_url}/update_weights_from_disk", json={"model_path": abs_path}, timeout=self.timeout)
    ok = resp.status_code == 200

Defensive patterns

Strategy: try-catch

Validate before calling

# Pre-flight before update_policy(): server up + shared path visible + writable
assert engine.health(), 'SGLang server not healthy'
shared = os.path.abspath(engine.shared_ckpt_path)
assert os.path.isdir(shared), f'shared path missing on trainer: {shared}'
probe = os.path.join(shared, '.probe')
with open(probe, 'w') as f: f.write('x')
os.remove(probe)  # trainer side writable; confirm same volume is mounted in the server container too

Try / catch

try:
    ok = engine.update_policy()
except RuntimeError as e:
    if 'SGLang update_policy failed' in str(e):
        # the real cause was printed as '[SGLANG WARNING] update_weights ...' on rank 0;
        # flush stale cache and retry once before aborting the RL step
        engine.flush_cache()
        ok = engine.update_policy()
        if not ok:
            raise
    else:
        raise

Prevention

Health-check the SGLang server (GET /health) at the start of every training run, not just at update time.
Use a genuinely shared volume (same host path or correctly mounted NFS) for the checkpoint dir; test write+read from both trainer and server containers.
Save checkpoints atomically (temp dir + rename) so the server never loads partially written weights.
Set the update timeout proportional to model size; a timeout manifests as this same RuntimeError.
Keep trainer and SGLang versions in lockstep so /update_weights_from_disk semantics do not drift.
Treat the printed WARNING text as part of your monitoring — it carries resp.status_code and resp.text from SGLang.

When it happens

Trigger: Calling update_policy() while training multi-process (dist.is_initialized) after a gradient step: rank 0 writes the checkpoint to sglang_shared_path and calls POST {base_url}/update_weights_from_disk with json {'model_path': abs_path}. It fails when the SGLang server is down/unreachable, when the path is not visible to the server process (different mount/container), when save_pretrained partially wrote (no atomic rename), when the server rejects the non-safetensors pickle format, or on update timeout (self.timeout). Non-200 prints a WARNING with resp.text; exceptions print the exception; both set ok=False which is broadcast, then raise.

Common situations: SGLang server and trainer in separate containers/NFS mounts where the 'shared' checkpoint dir is not actually shared; SGLang version that lacks or renamed /update_weights_from_disk; disk full or permission error while writing the checkpoint; race where the server reads weights mid-write because safe_serialization=False writes a pickle non-atomically; timeout too small for large models; rank>0 processes raising because rank 0 failed and the broadcast propagated ok=0.

AI-assisted analysis of jingyaogong/minimind@393e387e9a (2026-08-15). Data as JSON: /api/errors/a5532e118ab735a6. Report an issue: GitHub.