{"record":{"id":"fe965af8c7e6afca","repo":"sgl-project/sglang","slug":"the-memory-capacity-is-unbalanced-some-gpus-may-b","errorCode":null,"errorMessage":"The memory capacity is unbalanced. Some GPUs may be occupied by other processes. {pre_model_load_memory=}, {local_gpu_memory=}, {local_gpu_memory * 0.9=}","messagePattern":"The memory capacity is unbalanced\\. Some GPUs may be occupied by other processes\\. (.+?), (.+?), (.+?)","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/distributed/bootstrap.py","lineNumber":345,"sourceCode":"    logger.info(\n        \"TP LM-head PyNCCL all-to-all warmup completed in %.3fs \"\n        \"(tp_size=%d, bytes_per_peer=%d)\",\n        warmup_elapsed,\n        tp_group.world_size,\n        _TP_ALL_TO_ALL_WARMUP_BYTES_PER_PEER,\n    )\n\n\ndef _check_tp_memory_balance(\n    *, pre_model_load_memory: float, local_gpu_memory: float\n) -> None:\n    if pre_model_load_memory < local_gpu_memory * 0.9:\n        msg = \"The memory capacity is unbalanced. Some GPUs may be occupied by other processes. \"\n        msg += (\n            f\"{pre_model_load_memory=}, {local_gpu_memory=}, {local_gpu_memory * 0.9=}\"\n        )\n        if envs.SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK.get():\n            raise RuntimeError(msg)\n        else:\n            logger.warning(msg)\n","sourceCodeStart":327,"sourceCodeEnd":348,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/distributed/bootstrap.py#L327-L348","documentation":"During TP bootstrap, _check_tp_memory_balance compares free GPU memory before model load across ranks; if a rank's pre_model_load_memory is below 90% of the highest local_gpu_memory, some GPU is likely occupied by another process. It raises RuntimeError (only when SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK is enabled) or logs a warning otherwise.","triggerScenarios":"init_torch_distributed with tensor parallelism where one rank has >10% less free memory than the max — e.g. a leftover process, a second server, MPS, or Jupyter kernel holding memory on one GPU of the TP group.","commonSituations":"Stale sglang/vllm processes not fully killed before relaunch; sharing GPUs with other jobs; zombie CUDA contexts after a crash; inconsistent GPU SKUs or MIG configurations in one group.","solutions":["Kill leftover GPU processes (nvidia-smi, then kill/fuser on the offending PID) and relaunch","Set SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 (or false) to downgrade to a warning if the imbalance is intentional","Ensure all GPUs in the TP group are identical and exclusively available to this deployment"],"exampleFix":"# before\nGPU 0: 12 GB used by stale process -> RuntimeError at bootstrap\n# after\nkill -9 <stale_pid>  # GPU 0 free, bootstrap passes","handlingStrategy":"validation","validationCode":"import torch\nfree, _ = torch.cuda.mem_get_info()\n# compare across ranks; ensure min_free >= 0.9 * max_free before init_torch_distributed","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Run nvidia-smi and kill stale GPU processes before launching TP servers","Keep SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK off unless you want hard failure on imbalance","Reserve GPUs exclusively per deployment"],"tags":["distributed","tp","gpu-memory","resource-conflict"],"backgroundTag":"gpu-memory-unbalanced","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}