sgl-project/sglang · error · RuntimeError

[weight_cache] {op} of model weights is not supported while

Error message

[weight_cache] {op} of model weights is not supported while the weight cache is active (--weight-cache-mode {mode}): the weights are shared with the daemon via CUDA IPC, so freeing them would corrupt the daemon's master copy and every co-attached engine. Restart with --weight-cache-mode off to use this operation.

What it means

With --weight-cache-mode active, model weights are shared with a weight-cache daemon via CUDA IPC, so freeing/re-reserving them would corrupt the daemon's master copy and peer engines. The weight updater blocks release_memory_occupation/resume_memory_occupation in that mode.

Source

Thrown at python/sglang/srt/managers/scheduler_components/weight_updater.py:203

                self.record_weight_version_after_update(recv_req.weight_version)
            else:
                logger.error(message)
            torch.distributed.barrier(group=self.tp_cpu_group)
            return UpdateWeightsFromIPCReqOutput(success=success, message=message)

    def get_weights_by_name(self, recv_req: GetWeightsByNameReqInput):
        parameter = self.tp_worker.get_weights_by_name(recv_req)
        return GetWeightsByNameReqOutput(parameter=parameter)

    def _assert_weight_cache_inactive(self, op: str) -> None:
        """Reject freeing/restoring model weights while the CUDA IPC weight
        cache is active: the weights are shared with the daemon via CUDA IPC, so
        freeing them would leave the daemon and every peer pointing at released
        memory.
        """
        mode = self.tp_worker.model_runner.server_args.weight_cache_mode
        if mode != "off":
            raise RuntimeError(
                f"[weight_cache] {op} of model weights is not supported while the "
                f"weight cache is active (--weight-cache-mode {mode}): the weights "
                f"are shared with the daemon via CUDA IPC, so freeing them would "
                f"corrupt the daemon's master copy and every co-attached engine. "
                f"Restart with --weight-cache-mode off to use this operation."
            )

    def release_memory_occupation(self, recv_req: ReleaseMemoryOccupationReqInput):
        assert (
            self.is_fully_idle()
        ), "release_memory_occupation should be called only when server is idle."

        tags = recv_req.tags

        if tags is None or len(tags) == 0:
            tags = GPU_MEMORY_ALL_TYPES

        for tag in tags:

View on GitHub (pinned to 0132848349)

Solutions

1. Restart with --weight-cache-mode off if you need the memory-release flow
2. Avoid sleep/hibernate or flows that call release_memory_occupation while weight cache is on
3. Gate your orchestration: skip release operations when weight_cache_mode != 'off'

Example fix

# before
python -m sglang.launch_server --weight-cache-mode daemon --enable-sleep-mode
# after
python -m sglang.launch_server --weight-cache-mode off --enable-sleep-mode

mode = server_args.weight_cache_mode
assert mode == 'off' or not needs_memory_release_flow, 'weight cache blocks release_memory_occupation'

When it happens

Trigger: Calling release_memory_occupation or resume_memory_occupation (e.g. sleep-mode / memory hibernation flows, some update-weight or restart flows) while weight_cache_mode != 'off'.

Common situations: Enabling the weight cache daemon for fast multi-engine startup and then triggering a flow that releases GPU memory (server sleep, weight updates requiring re-init).

Related errors

• QVGPackedCausalKVCache does not support pinned-sink (longliv
• Multi-node weight cache daemons (nnodes > 1) require --dist-
• Weight cache daemon for pp_rank={pp_rank} tp_rank={tp_rank}
• Weight cache daemon (pid={p.pid}) exited prematurely with co
• additional customized generation output is not supported by

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0cf029d7140f6548. Report an issue: GitHub.