sgl-project/sglang · error · RuntimeError

The memory capacity is unbalanced. Some GPUs may be occupied

Error message

The memory capacity is unbalanced. Some GPUs may be occupied by other processes. {pre_model_load_memory=}, {local_gpu_memory=}, {local_gpu_memory * 0.9=}

What it means

During TP bootstrap, _check_tp_memory_balance compares free GPU memory before model load across ranks; if a rank's pre_model_load_memory is below 90% of the highest local_gpu_memory, some GPU is likely occupied by another process. It raises RuntimeError (only when SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK is enabled) or logs a warning otherwise.

Source

Thrown at python/sglang/srt/distributed/bootstrap.py:345

    logger.info(
        "TP LM-head PyNCCL all-to-all warmup completed in %.3fs "
        "(tp_size=%d, bytes_per_peer=%d)",
        warmup_elapsed,
        tp_group.world_size,
        _TP_ALL_TO_ALL_WARMUP_BYTES_PER_PEER,
    )


def _check_tp_memory_balance(
    *, pre_model_load_memory: float, local_gpu_memory: float
) -> None:
    if pre_model_load_memory < local_gpu_memory * 0.9:
        msg = "The memory capacity is unbalanced. Some GPUs may be occupied by other processes. "
        msg += (
            f"{pre_model_load_memory=}, {local_gpu_memory=}, {local_gpu_memory * 0.9=}"
        )
        if envs.SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK.get():
            raise RuntimeError(msg)
        else:
            logger.warning(msg)

View on GitHub (pinned to 0132848349)

Solutions

1. Kill leftover GPU processes (nvidia-smi, then kill/fuser on the offending PID) and relaunch
2. Set SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 (or false) to downgrade to a warning if the imbalance is intentional
3. Ensure all GPUs in the TP group are identical and exclusively available to this deployment

Example fix

# before
GPU 0: 12 GB used by stale process -> RuntimeError at bootstrap
# after
kill -9 <stale_pid>  # GPU 0 free, bootstrap passes

import torch
free, _ = torch.cuda.mem_get_info()
# compare across ranks; ensure min_free >= 0.9 * max_free before init_torch_distributed

When it happens

Trigger: init_torch_distributed with tensor parallelism where one rank has >10% less free memory than the max — e.g. a leftover process, a second server, MPS, or Jupyter kernel holding memory on one GPU of the TP group.

Common situations: Stale sglang/vllm processes not fully killed before relaunch; sharing GPUs with other jobs; zombie CUDA contexts after a crash; inconsistent GPU SKUs or MIG configurations in one group.

Related errors

• --enable-tp-lm-head-all-to-all requires an available PyNCCL
• world_size must be positive and divide global_heads
• Group {group_name} is destroyed.
• world_size ({world_size}) is less than tensor_parallel_degre
• {type(self).__name__} does not implement ring KV-chunk atten

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fe965af8c7e6afca. Report an issue: GitHub.